Quality Decision
Framework

The Three Decisions

After evaluating AI output, you have three choices:

Accept: The output meets your standards. Use it as-is. Done.

Refine: The output is close. Give feedback to the AI to improve it. Iterate until it meets standards.

Restart: The output is not salvageable. Start over with a different prompt, different AI system, or different approach entirely.

The goal of this chapter is to give you a framework for making these decisions consistently and wisely.

Risk Calibration

Not All Outputs Are Created Equal

The decision criteria are different depending on the stakes of the output. A brainstorm document has much lower stakes than a hiring decision, a legal contract, or a medical recommendation. You need to calibrate your standards to match the risk level.

The Risk-Standards Matrix

Low Risk (Creative, brainstorms, internal drafts): Good enough is good enough. If it is 80% what you wanted, great. Move forward. Perfect is not required.

Medium Risk (Customer-facing content, internal documentation, competitive analysis): Good quality is required. You need 90%+ alignment with your standards. Accuracy needs spot-checking. Bias needs checking. But perfection is not required.

High Risk (Legal documents, medical recommendations, hiring decisions, financial advice): Excellent quality is required. You need near-perfect alignment. Multiple dimensions must be checked thoroughly. Human expert review is required for final decision-making.

Criteria for Accepting

Accuracy: For the level of risk, how accurate does it need to be? Critical facts must be correct. Minor details can be wrong if the overall direction is right.

Completeness: Does it cover the necessary ground? Would using this output as-is create gaps or missing information that would cause problems?

Relevance: Does it actually address your need? Is it focused or off-track? Would a user understand that this answers their question?

Appropriateness: Is the tone, format, and style acceptable for its use case? Would you be comfortable sharing this with your intended audience?

Alignment with Standards: How does this align with your standards for that risk level? For low-risk work, 80% alignment might be acceptable. For high-risk work, you need 95%+.

Criteria for Refining

Refine when:

• The output is close (80-95% aligned) but has specific gaps or issues that targeted feedback can fix
• The problem is fixable without starting over (e.g., "make it more formal" or "add more examples")
• The time cost of refining is less than the cost of starting over
• You are confident that iteration will improve it (not just spinning wheels)

When NOT to Refine

Do not refine when:

• The output is fundamentally misaligned with what you need (below 70%)
• You have already iterated 4-5 times without major improvement
• The effort to refine exceeds the value of getting a slightly better result
• The time constraints do not allow for iteration

Criteria for Restarting

Restart when:

• The output is fundamentally not useful (far below your standards)
• The AI system misunderstood what you were asking in ways that refinement would not fix
• The wrong approach is being taken and no amount of feedback will fix it
• You have clear ideas for a better way to prompt or a better tool to use

Decision Tree Framework

Here is a systematic way to make the decision:

Step 1: Quick Scan Does the output seem directionally correct? If no, probably restart. If yes, continue to step 2.

Step 2: Full Evaluation Apply ACRE (Accuracy, Completeness, Relevance, Appropriateness) and score each dimension.

Step 3: Calculate Alignment Percentage What percentage of the output meets your standards? (Average the four dimensions.)

Step 4: Compare to Risk Level For your risk level, what is the minimum acceptable alignment percentage? (Low risk: 70%, Medium: 85%, High: 95%)

Step 5: Decide

• If alignment is above your minimum: Accept (or refine if minor gaps are identified)
• If alignment is 10-20% below your minimum: Refine
• If alignment is more than 20% below your minimum: Restart

Cost-Benefit Analysis for Refinement

Sometimes the decision is not just about quality. It is about effort. Ask yourself:

Cost of refining: How many iterations will it take? How much time? Is the output getting closer or stuck?

Benefit of refining: If I refine this, how much better will it be? Is the improvement worth the cost?

Cost of accepting imperfect: If I accept this output despite its issues, what is the risk? What could go wrong?

Cost of starting over: How much time will it take to write a better prompt and get a better result?

Make a conscious trade-off decision. Sometimes accepting 85% quality is better than spending 3 hours for 90%. Sometimes it is not.

Context-Specific Guidance

Creative Writing & Content

Accept when the piece is engaging and on-brand, even if not perfect. The reader values voice and personality over perfection. Refine if tone is off or there are gaps. Restart if the whole direction is wrong.

Technical Documentation

Accept when it is accurate and complete enough for its audience. Refine if there are clarity issues. Restart if technical details are wrong.

Business Analysis & Strategy

Accept when the recommendations are sound and well-reasoned. Refine if analysis is incomplete or missing key considerations. Restart if fundamental assumptions are wrong.

Code Generation

Accept when it works, is readable, and handles edge cases. Refine if there are style issues. Restart if there are bugs or architectural problems.

Customer-Facing Communication

Accept when the message is clear and appropriate. Refine if tone is off. Restart if it could damage relationships or brand.

Documentation for Decision

For important decisions, document your reasoning:

What was evaluated: What quality dimensions did you check?

What you found: What are the strengths and weaknesses?

Your decision: Accept, refine, or restart?

Your reasoning: Why did you make this decision?

This documentation serves multiple purposes: it helps you make consistent decisions over time, it creates accountability, it helps others understand your standards, and it creates a record if there are questions later.

Final Thoughts: Evaluation as a Practice

Critical evaluation of AI output is a skill. Like all skills, it improves with practice. The frameworks in this lesson are tools to help you think systematically. Over time, you will develop intuition and you will not need to work through every step. But even experienced evaluators benefit from occasionally returning to systematic frameworks to catch blind spots.

The goal is not perfection. The goal is responsible use of AI. Make decisions consciously. Know why you accepted something or rejected it. Build a track record of good decisions. Hold systems accountable when they produce poor results. This is how you become trustworthy with AI systems.