16 Evaluation, Testing & Optimization Practice Questions & Answers

1. 1. A team is defining success criteria for a sentiment-analysis feature powered by Claude. According to Anthropic's guidance on defining success, which of the following is the BEST-written success criterion?

   • A."Users should generally feel satisfied with the feature"
   • B."The model should never make a mistake on any input"
   • C."Sentiment classification achieves an F1 score of at least 0.85 on the held-out evaluation set"✓ Answer
   • D."The model should show good performance on customer messages"

   Anthropic's docs say good success criteria must be specific, measurable, achievable, and relevant. An F1 target of 0.85 on a defined eval set is specific and measurable, and it is achievable for current models; "good performance" and "satisfied users" are vague and unmeasurable, while "never make a mistake" is unachievable and therefore useless as a target.

   Source: Claude Docs, Test & evaluate — "Define your success criteria": good criteria are specific, measurable, achievable, relevantReport a problem with this question

2. 2. A support chatbot passes its accuracy evals before launch, but after release users abandon it because responses are slow and per-conversation API spend is far over budget. Per Anthropic's evaluation guidance, what was the core flaw in the eval framework?

   • A.Accuracy is the wrong metric for any chatbot and should never be tracked
   • B.It used automated grading instead of human grading
   • C.The eval dataset was too large, which inflated the accuracy score
   • D.It measured only one dimension instead of multidimensional criteria covering latency and cost alongside quality✓ Answer

   The docs state that most use cases need multidimensional evaluation across several success criteria — for example task fidelity plus operational metrics like response time and price per call (e.g., "95% of responses under a latency target"). Because latency and cost were never part of the criteria, the suite could not catch failures on those dimensions; grading method and dataset size are unrelated to this gap.

   Source: Claude Docs, "Define your success criteria" — multidimensional evaluation (task fidelity, latency, price)Report a problem with this question

3. 3. When building an evaluation suite before launch, which approach does Anthropic's eval design guidance recommend?

   • A.Only free-form open-ended questions, since structured questions oversimplify the task
   • B.Skipping evals until real user traffic arrives, since synthetic cases are unrepresentative
   • C.A large volume of automatically gradable test cases, even if each has slightly lower signal than hand grading✓ Answer
   • D.A small set of test cases graded carefully by hand, since human judgment is the gold standard

   Anthropic's eval design principles are: be task-specific, automate when possible, and prioritize volume over quality — explicitly, "more questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals." Automated grading at scale lets you iterate quickly, which small hand-graded sets cannot support.

   Source: Claude Docs, "Create strong empirical evaluations" — Eval design principles: prioritize volume over qualityReport a problem with this question

4. 4. A developer builds an eval dataset for a document Q&A app using only clean, well-formed questions about content that exists in the documents. Per Anthropic's guidance on being task-specific, what is the main weakness of this dataset?

   • A.It omits edge cases such as ambiguous questions, overly long inputs, and questions about nonexistent or irrelevant data✓ Answer
   • B.Questions about existing content are too easy and should be removed entirely
   • C.It should contain only multiple-choice questions so grading is objective
   • D.Eval datasets must be generated exclusively by the model under test

   The docs instruct you to design evals that mirror your real-world task distribution and to factor in edge cases, including irrelevant or nonexistent input data, overly long inputs, poor or harmful user input, and ambiguous cases. A dataset with only clean answerable questions can't reveal how the app behaves on exactly the inputs most likely to cause production failures such as hallucinated answers.

   Source: Claude Docs, "Create strong empirical evaluations" — Be task-specific: include edge casesReport a problem with this question

5. 5. In a mixed-methodology test framework, which task is the BEST fit for code-graded (programmatic) evaluation rather than model-graded or human-graded evaluation?

   • A.Assessing whether a summary faithfully captures the nuance of a legal brief
   • B.Judging the empathy and professionalism of customer-service replies
   • C.Deciding whether a chatbot's tone matches the brand voice
   • D.Checking that a sentiment classifier outputs exactly one of the labels positive, negative, or neutral, matching the expected label✓ Answer

   Code-based grading (e.g., exact string match after normalization) is the fastest and most reliable method but the least flexible, so it fits tasks with categorical, deterministic answers like a fixed label set. Empathy, nuance, and tone are subjective qualities that require LLM-based or human grading because no simple program can score them.

   Source: Claude Docs, "Create strong empirical evaluations" — grading methods: code-based grading is fastest/most reliable, least flexible; exact match for categorical tasksReport a problem with this question

6. 6. According to Anthropic's comparison of grading methods, what is the main tradeoff of human grading in an eval pipeline?

   • A.It scales automatically with the size of the eval dataset
   • B.It is the most flexible and highest quality, but slow and expensive, so it should be used sparingly✓ Answer
   • C.It is faster than code-based grading but cannot handle subjective criteria
   • D.It is the cheapest method but produces the least reliable scores

   The docs characterize human grading as the most flexible and high-quality method — it can judge anything a person can judge — but also slow and expensive, advising teams to avoid it where possible. That is why the recommended pattern is automated grading (code- or LLM-based) for volume, with humans reserved for calibration or small high-stakes samples.

   Source: Claude Docs, "Create strong empirical evaluations" — grading methods: human grading is most flexible but slow and expensive; avoid if possibleReport a problem with this question

7. 7. A team uses an LLM judge to grade chatbot responses for professionalism on a 1-5 Likert scale, but the scores are noisy and inconsistent. Which change follows Anthropic's tips for reliable LLM-based grading?

   • A.Remove all instructions so the judge scores purely on intuition
   • B.Raise the judge's temperature so it explores more diverse scores
   • C.Have the judge grade its own reasoning multiple times and always keep the highest score
   • D.Give the judge a detailed rubric with concrete criteria, ask it to reason before deciding, and constrain the final output to just the grade✓ Answer

   Anthropic's tips for LLM-based grading are to provide detailed, clear rubrics, keep grading empirical (based on stated criteria rather than vibes), and encourage the judge to reason through its evaluation before emitting a constrained final answer such as a single number. A rubric anchors the scale so repeated runs converge, whereas higher temperature or no instructions increases variance.

   Source: Claude Docs, "Create strong empirical evaluations" — Tips for LLM-based grading: detailed rubrics, encourage reasoning, constrained outputReport a problem with this question

8. 8. An eval computes sentence embeddings for the model's answers to several paraphrases of the same question and compares them with cosine similarity. Which success criterion is this eval measuring?

   • A.Safety: whether the answers avoid toxic content
   • B.Latency: how quickly the model answers each paraphrase
   • C.Consistency: whether similar inputs yield semantically similar answers✓ Answer
   • D.Cost: how many tokens each answer consumes

   In the docs' example evals, cosine similarity between embeddings — where values near 1 indicate high semantic similarity — is used to measure consistency: paraphrased versions of a question should produce semantically similar answers. Embedding similarity says nothing about speed, toxicity, or token spend, so the other criteria would need different instruments.

   Source: Claude Docs, "Create strong empirical evaluations" — example eval: consistency via cosine similarity of embeddingsReport a problem with this question

9. 9. Which automated metric does Anthropic's eval guidance illustrate for grading summarization quality against a reference summary?

   • A.The compression ratio of summary length to document length
   • B.Perplexity of the candidate summary under the judge model
   • C.ROUGE-L, based on the longest common subsequence between the candidate and reference summaries✓ Answer
   • D.Levenshtein edit distance between the summary and the full source document

   The docs' example evals use ROUGE-L, which scores the longest common subsequence shared by the candidate and a reference summary, as an automated measure of summarization quality and coverage. Edit distance to the full document and compression ratio don't compare against a quality reference, and perplexity measures fluency rather than content overlap.

   Source: Claude Docs, "Create strong empirical evaluations" — example eval: summarization graded with ROUGE-L (longest common subsequence)Report a problem with this question

10. 10. A team wants to know whether a rewritten system prompt actually improves their document Q&A app. Which procedure is the soundest A/B test of the new prompt?

    • A.Run both prompt versions on the same eval dataset with the same model, settings, and graders, changing only the prompt, then compare metrics✓ Answer
    • B.Run the new prompt on a fresh, different eval dataset so results aren't biased by the old data
    • C.Ask a few teammates to chat with the new version and vote on whether it feels better
    • D.Switch the new prompt to a more capable model at the same time to maximize the visible improvement

    A valid A/B comparison isolates a single variable: holding the eval dataset, model, parameters, and grading method constant means any metric difference is attributable to the prompt change alone. Changing the dataset or the model at the same time confounds the comparison, and informal chats lack the volume and consistency that empirical evaluation requires.

    Source: CCAR-P Exam Guide Domain 4; Claude Docs "Define your success criteria" — A/B testing as a quantitative measurement method (single-variable comparison)Report a problem with this question

11. 11. After a company refreshes the knowledge base behind its Claude-powered assistant, users report answers that sound confident but are wrong, citing details that appear nowhere in the new documents. What is the most likely diagnosis and first remediation?

    • A.Model mismatch: immediately migrate to the largest available model, since bigger models never hallucinate
    • B.An eval dataset problem: delete the failing test cases so the suite passes again
    • C.A latency problem: enable streaming so wrong answers appear faster and users can correct them
    • D.Hallucination from ungrounded answers: require the model to extract supporting quotes from the retrieved documents and explicitly allow it to say "I don't know," then re-run the eval suite✓ Answer

    Confident answers containing details absent from the source documents are the signature of hallucination, not slowness or grader error. Anthropic's hallucination-reduction guidance is to ground responses in verifiable evidence — for example, have the model extract relevant quotes from the provided documents before answering — and to give it explicit permission to admit uncertainty, since forcing an answer on every query pushes the model to fabricate.

    Source: Claude Docs, Strengthen guardrails — "Reduce hallucinations": allow the model to say "I don't know"; ground answers in extracted quotesReport a problem with this question

12. 12. A classification prompt has clear instructions, well-chosen few-shot examples, and chain-of-thought guidance, yet a small fast model still fails consistently on cases that require multi-step legal reasoning. Which conclusion best fits this evidence?

    • A.It is an infrastructure bug: retry the same requests until they pass
    • B.It is hallucination: the model is inventing facts and needs quote grounding
    • C.It is a prompt failure: the instructions must still be ambiguous somewhere
    • D.It is likely model mismatch: the task's complexity exceeds the model's capability, so evaluate a more capable model on the same eval suite✓ Answer

    When prompt-engineering levers (clear instructions, examples, chain of thought) are exhausted and failures remain consistent on the hardest reasoning cases, the bottleneck is the model's capability rather than the wording — the standard remedy is to trade up to a more capable model and confirm the gain on the same eval suite. Prompt failures typically show up as inconsistent or format-related errors that improve when instructions are clarified, which is not the pattern here.

    Source: CCAR-P Exam Guide Domain 4 — diagnosing prompt failure vs. model capability mismatch; Claude Docs model-selection guidance (match model capability to task complexity)Report a problem with this question

13. 13. Users of a Claude-powered writing assistant complain that the app "feels frozen" while long responses are generated, even though total generation time is normal for the output length. Which optimization most directly addresses the perceived latency without changing the model or shortening the output?

    • A.Send every request through a batch-processing endpoint
    • B.Increase max_tokens so the model has more room to finish
    • C.Enable streaming so tokens are displayed to the user as they are generated✓ Answer
    • D.Raise the temperature so the model writes faster

    Anthropic's latency guidance lists streaming as a key technique: it delivers tokens incrementally, so the time to first visible token drops dramatically and the interface no longer looks frozen, even though total generation time is unchanged. Temperature does not control generation speed, a larger max_tokens does nothing for responsiveness, and batch processing trades latency away for throughput — the opposite of what interactive users need.

    Source: Claude Docs, Strengthen guardrails — "Reduce latency": leverage streaming; streaming improves perceived responsivenessReport a problem with this question

14. 14. A customer-support app prepends the same large, rarely-changing policy manual to every API request, and input tokens dominate the bill. Which optimization most directly cuts this cost while keeping the manual available to the model?

    • A.Split the manual across several parallel requests and merge the answers
    • B.Raise the temperature so the model needs less context to answer
    • C.Use prompt caching so the static manual prefix is cached and repeated requests read it at a reduced rate✓ Answer
    • D.Move the manual from the system prompt into every user message

    Prompt caching exists precisely for this pattern: a large, stable prefix (like a policy manual) is written to the cache once, and subsequent requests that reuse the identical prefix are billed at a much lower cache-read rate, cutting both cost and time-to-first-token. Moving the text between message roles doesn't reduce tokens, temperature has no effect on context requirements, and splitting the manual multiplies requests instead of saving them.

    Source: Claude Docs — Prompt caching: cached static prompt prefixes are read at a reduced rate, lowering cost and latency for repeated contextReport a problem with this question

15. 15. A new, more capable Claude model becomes available and a team wants to upgrade their production assistant. What is the recommended regression-testing practice before switching?

    • A.Spot-check five favorite prompts manually and switch if they look fine
    • B.Discard the old eval suite and write a brand-new one, since old test cases can't apply to a new model
    • C.Run the full existing eval suite against the new model with prompts held fixed, compare results to the current model's baseline, and investigate any regressions before cutover✓ Answer
    • D.Switch production immediately, since a newer model is always at least as good on every task

    An eval suite's core value is exactly this: it is a fixed, repeatable benchmark, so re-running it with only the model changed isolates the upgrade's effect and surfaces task-specific regressions that aggregate benchmarks hide — newer models can behave differently on your particular distribution even when generally stronger. Assuming improvement, discarding the baseline, or spot-checking a handful of prompts all forfeit the controlled comparison that catches regressions before users do.

    Source: CCAR-P Exam Guide Domain 4 — regression testing across model upgrades; Claude Docs eval methodology (fixed eval suite as baseline for controlled comparison)Report a problem with this question

16. 16. Which logging and observability strategy best supports diagnosing production failures and iteratively improving a Claude-powered application?

    • A.Store only weekly averages of latency to minimize storage costs
    • B.Log only requests that return HTTP errors, since successful responses cannot be wrong
    • C.Log full prompt/response pairs (handled per data-privacy policy) with per-request token usage, latency, and error rates, so failures can be traced and real traffic can seed new eval cases✓ Answer
    • D.Avoid logging model outputs entirely, because outputs are nondeterministic and therefore not useful

    LLM failures such as hallucinations return successful HTTP 200 responses, so error-only logging misses exactly the failures that matter; you need the full prompt and output to reproduce and diagnose a bad answer, plus per-request token, latency, and error telemetry to track cost and performance criteria. Captured production failures are also the best raw material for growing the eval dataset, closing the iterative-improvement loop; weekly averages hide tail latency and individual failures.

    Source: CCAR-P Exam Guide Domain 4 — logging and observability for LLM applications; Claude Docs eval guidance (production traffic informs eval datasets; track operational metrics)Report a problem with this question

Practice questions based on the official Claude Certified Architect – Professional (CCAR-P) exam guide and Anthropic's public documentation. This is an independent study tool, not affiliated with or endorsed by Anthropic, and does not grant certification. The real exam is 63 questions, 120 minutes, passing at a scaled 720/1000, delivered via Pearson VUE ($175). Official certification page →