22 Evaluation, Testing & Optimization Practice Questions & Answers

1. 1. A team is about to start improving a Claude-powered claims-summarisation feature. They have a backlog of user complaints but no evaluation set, and the lead proposes tuning prompts now and building evals once the prompt settles. Select 1: what is the strongest objection to that order?

   • A.Without provider registration of an evaluation harness, prompt changes cannot deploy at all, making evals a hard technical prerequisite.
   • B.Without a model-tier decision on the table an evaluation set has no use, so it belongs after the prompt work rather than before it.
   • C.Without a settled prompt there is nothing for the model to generate the evaluation dataset from, so the sequence has to start there.
   • D.Without a fixed evaluation set, each prompt change is judged on ad-hoc examples, so a genuine gain cannot be told apart from noise or a new regression.✓ Answer

   An evaluation set is the measurement instrument, and one built after the changes have been made is shaped by them. A fixed, versioned set built before optimisation lets every candidate change be scored against the same tasks, so gains and regressions become visible rather than assumed. Nothing in the platform requires registering an eval harness, and eval tasks come from real traffic rather than from the prompt.

   Source: Anthropic, "Create strong empirical evaluations" (platform.claude.com/docs/en/test-and-evaluate/develop-tests); CCAR-P Domain 4 (Evaluation, Testing & Optimization)Report a problem with this question

2. 2. A support-triage assistant scores well in demos but misroutes real tickets. The team is assembling its first evaluation set of about 30 tasks. Select 1: which composition best represents production?

   • A.Clean, well-formed tickets written by the team so every task carries an unambiguous label and messy input does not drag the score down.
   • B.Real production tickets weighted toward the categories and edge cases that have actually produced misroutes, with positive and negative examples.✓ Answer
   • C.Synthetic tickets generated by Claude from the routing policy document, which covers every documented category evenly and can be regenerated cheaply.
   • D.Tickets from the highest-volume category only, since fixing the common path moves the aggregate accuracy number more than anything else.

   A representative set is mined from real traffic and deliberately over-samples the hard cases and the failure modes already observed, because those are what the change is meant to fix; balancing positive and negative cases keeps the score honest. Clean, single-category or purely synthetic sets measure a distribution the system never faces in production.

   Source: Anthropic Engineering, "Demystifying evals for AI agents" (20-50 tasks mined from real failures); CCAR-P Domain 4Report a problem with this question

3. 3. An invoice-extraction pipeline returns a JSON object with vendor, date and total for each document. The team must grade 400 evaluation tasks nightly. Select 1: which grading approach fits this task?

   • A.Cosine similarity between the model output and the reference record, thresholded so near-identical extractions pass and different ones fail.
   • B.Code-based grading that parses the JSON and compares each field to the labelled value, with normalisation for date and currency formatting.✓ Answer
   • C.An LLM judge scoring each extraction 1-5 for overall quality, since a judge tolerates the formatting variation that string equality would fail on.
   • D.Human review of a 20-task sample each night, treated as the primary score because extraction errors carry real financial consequences.

   When the correct answer is checkable — a field either matches the labelled value or it does not — a deterministic grader is the fastest, most reproducible and cheapest option, and normalisation absorbs the valid formatting variation that naive string equality would fail on. Judges and similarity scores introduce non-determinism where none is needed, and human review cannot cover 400 tasks nightly.

   Source: Anthropic, "Create strong empirical evaluations" — code-based grading preferred where outputs are exactly checkable; CCAR-P Domain 4Report a problem with this question

4. 4. A team grades a legal-drafting assistant on faithfulness to source, tone and completeness. Their judge prompt asks one model call to weigh all three and emit a single 1-5 score, and quality debates keep stalling. Select 1: what should change?

   • A.Run one judge per dimension, each with its own rubric and its own call, so faithfulness, tone and completeness produce separate scores.✓ Answer
   • B.Keep the single holistic judge but raise the scale to 1-10, giving it the resolution needed to express trade-offs between the three dimensions.
   • C.Keep the single holistic call and add few-shot exemplars of 1, 3 and 5 outputs so the judge anchors its overall scale more consistently.
   • D.Replace the judge with the generating model itself at higher effort, since the author has full context on which dimension it compromised.

   A holistic judge silently averages dimensions, so a drop in faithfulness can be hidden by good tone and the resulting number is not actionable. One rubric and one isolated call per dimension yields separate, diagnosable signals; a finer scale or few-shot anchors do not undo the averaging, and using the generator as its own judge removes the independence the grade depends on.

   Source: Anthropic, "Create strong empirical evaluations" — LLM-based grading guidance (one rubric per dimension); CCAR-P Domain 4Report a problem with this question

5. 5. An LLM judge scores retrieval answers as grounded or not. Its scores drift week to week even on unchanged outputs, and reviewers cannot reproduce them. Select 1: which judge design most improves trustworthiness?

   • A.A grading prompt regenerated each run from the current rubric text, so the judge always reflects the newest definition of groundedness the team holds.
   • B.The same model and prompt as the generator, so the judge shares its interpretation of the task and stops disagreeing with the answer under review.
   • C.A fixed, versioned grading prompt that forces brief reasoning first, then a constrained verdict, with an 'unknown' escape hatch and periodic human spot-checks.✓ Answer
   • D.A higher sampling temperature on the judge, so its verdicts explore the space of readings and average out into a fairer aggregate score.

   Reproducibility comes from pinning the grader: a versioned prompt, forced reasoning that is then discarded, a constrained output format, and an explicit escape hatch instead of a forced guess — and then spot-checking the grader itself against human experts. Regenerating the prompt each run and raising temperature both add exactly the variance being complained about, and a judge that shares the generator's model and prompt inherits its blind spots.

   Source: Anthropic, "Create strong empirical evaluations" — LLM grader design (rubric, forced reasoning, constrained output, escape hatch); CCAR-P Domain 4Report a problem with this question

6. 6. A team has two senior clinicians available four hours a week and a nightly evaluation of 500 clinical-summary tasks. Select 1: how should that expert time be used?

   • A.Grade every task in the highest-risk category as the production score, with the automated grader relegated to the remaining lower-risk tasks.
   • B.Write additional evaluation tasks each week instead of grading, because authoring tasks widens coverage faster than grading them does.
   • C.Grade a stratified sample and compare it against the automated grader, adjusting the rubric wherever expert and grader verdicts diverge.✓ Answer
   • D.Grade the 40 tasks the automated grader already marks as passing, confirming the suite's headline pass rate before it goes to stakeholders.

   Human experts are the gold standard but cannot scale to 500 tasks a night, so their highest-leverage use is calibrating the automated grader — once calibrated, the grader carries the whole volume with a known error rate. Sampling only passing tasks or only one category leaves the grader's disagreement with experts unmeasured on exactly the cases that matter.

   Source: Anthropic, "Create strong empirical evaluations" — human grading as calibration for automated graders; CCAR-P Domain 4Report a problem with this question

7. 7. Before a pilot, a product owner writes the success criterion for a Claude-based chat deflection feature as 'the assistant answers correctly at least 90% of the time'. Select 1: what is the main defect in that criterion?

   • A.It commits to an accuracy number before an eval set exists, and success criteria can only be written after a baseline has been measured.
   • B.It is one-dimensional: accuracy is fixed while latency, cost per interaction and safety behaviour stay unconstrained and untracked.✓ Answer
   • C.It names a percentage rather than an absolute count, so the criterion cannot be checked until the pilot's total traffic is known.
   • D.It sets the bar too low: a deflection feature wrong one time in ten cannot survive contact with real production support volume.

   Strong success criteria are specific, measurable and multidimensional — an accuracy target stated alongside a latency percentile, a cost ceiling per interaction and a safety floor. A single accuracy number can be satisfied by a system that is too slow, too expensive or unsafe to ship, so on its own it cannot gate a launch.

   Source: Anthropic, "Define your success criteria" (platform.claude.com/docs/en/test-and-evaluate/define-success); CCAR-P Domain 4Report a problem with this question

8. 8. An agent books internal meeting rooms. The evaluation marks a trial as passing only if the agent calls search_rooms, then check_conflicts, then create_booking in that order, and trials that book the right room by another route are failing. Select 1: what is the fix?

   • A.Grade the end state — the booking that exists afterwards and its attributes — rather than the sequence of tool calls that produced it.✓ Answer
   • B.Replace the code grader with an LLM judge that reads the transcript and decides whether the trajectory the agent chose looks reasonable.
   • C.Broaden the accepted sequences to a whitelist of the three or four call orders reviewers agree are legitimate ways to reach a booking.
   • D.Keep the sequence check and add a system-prompt instruction pinning the call order, so behaviour matches what the grader already expects.

   Rigid path-checking punishes valid alternate approaches and turns the evaluation into a measure of conformity rather than capability. Verifying the end state — the right room booked, no double-booking, correct attendees — stays deterministic while admitting any correct route, whereas pinning the order in the prompt fixes the score by constraining the agent instead of measuring it better.

   Source: Anthropic Engineering, "Demystifying evals for AI agents" — grade outcomes and end state, not tool trajectories; CCAR-P Domain 4Report a problem with this question

9. 9. A summarisation team optimises against ROUGE-L overlap with reference summaries. Scores climb for six weeks while support agents report the summaries have become harder to use. Select 1: what does this most likely indicate?

   • A.The metric is a proxy rewarding surface overlap with references, so optimisation has drifted toward matching wording rather than the usefulness intended.✓ Answer
   • B.The reference summaries have degraded over the six weeks, so the score is measuring drift in the references rather than in the model output.
   • C.ROUGE-L is being computed on the wrong field, and the reported gains disappear once the summary body is scored instead of the full response.
   • D.The support agents are judging against a newer style guide than the one used when the reference summaries for the eval set were written.

   This is Goodhart's law in practice: ROUGE-L rewards n-gram overlap with a reference, which correlates with quality but is not quality, so sustained optimisation pushes the system toward the proxy and away from the goal. When proxy and goal diverge the fix is to re-anchor the metric — a rubric-based judge or a task-outcome measure reflecting what agents need — not to keep climbing the proxy.

   Source: Anthropic, "Create strong empirical evaluations" — choosing metrics that reflect the task goal; CCAR-P Domain 4Report a problem with this question

10. 10. A platform team keeps two evaluation suites. Suite A currently passes at 41% and decides whether a new approach is worth pursuing; suite B passes at 100% and runs on every prompt commit. Select 1: what is suite B for?

    • A.Calibration: its perfect score establishes the ceiling against which suite A's 41% is normalised into a directly comparable capability number.
    • B.Acceptance: a 100% suite is the gate stakeholders sign off on, and suite A is the internal research view that never leaves the team.
    • C.Regression: it holds near 100% by design, so any drop on a commit signals that a change has broken behaviour that previously worked.✓ Answer
    • D.Saturation control: a suite that reaches 100% is retired, so B is already scheduled for replacement by harder tasks drawn from suite A.

    Capability evaluations are designed to start low so they leave headroom to measure progress, while regression evaluations cover behaviour already known to work and are expected to sit at or near 100% — which is precisely what makes a single new failure an actionable alarm. A regression suite that starts failing is reporting degradation, not difficulty.

    Source: Anthropic Engineering, "Demystifying evals for AI agents" — capability vs regression evaluations; CCAR-P Domain 4Report a problem with this question

11. 11. A prompt edit fixes a reported failure on refund-eligibility questions. Select 1: what should happen before that edit ships?

    • A.Run the full regression suite on the edited prompt and compare per-task results with the previous version, not only the aggregate rate.✓ Answer
    • B.Ask a second reviewer to read the diff and confirm the new wording does not contradict any other instruction present in the system prompt.
    • C.Ship behind a flag and watch production complaint volume for a week, which measures the change on real traffic rather than on stale eval tasks.
    • D.Run the edit against the reported failure plus a handful of similar refund questions, since the change was scoped to that one behaviour.

    Prompt changes have non-local effects, because instructions interact and a fix for one case can quietly break others. Comparing per-task results between the two versions catches the tasks that flipped from pass to fail, which an unchanged or even improved aggregate pass rate can completely hide.

    Source: Anthropic, "Create strong empirical evaluations" — regression testing of prompt changes; CCAR-P Domain 4Report a problem with this question

12. 12. An engineer reports a 9-point evaluation gain from a change that swapped the model tier, rewrote the system prompt and doubled retrieved chunks in a single commit. Select 1: what is the correct response?

    • A.Keep the change but re-score it with a second LLM judge, because a swing that large usually reflects grader variance rather than a real gain.
    • B.Discard the result and rebuild the eval set, because a 9-point move on one commit indicates the tasks were too easy to be informative.
    • C.Accept the gain and lock the bundle in, because the combination is what will run in production and that is what the eval measured.
    • D.Re-run the three changes independently on the same held-out set, since a combined commit cannot attribute the gain to any one variable.✓ Answer

    Attribution requires isolating variables: with three simultaneous changes one may be carrying the entire gain while another is a regression the aggregate is hiding, and the team learns nothing reusable. Re-running each change against the same held-out set, with the eval version pinned alongside the prompt, is what turns a lucky commit into a repeatable finding.

    Source: Anthropic, "Create strong empirical evaluations" — controlled comparison and held-out test sets; CCAR-P Domain 4Report a problem with this question

13. 13. Finance compares two configurations of a document-QA agent. Config A costs less per API call but needs more turns to finish a task; config B costs more per call and usually finishes in one. Select 1: which comparison should decide the choice?

    • A.Cost per completed task, summing every call, retry and tool round-trip a task consumes end to end, since a cheaper call needing more turns is not cheaper.✓ Answer
    • B.Cost per successful first turn, which isolates the model's unaided quality from the harness retries that surround the underlying call.
    • C.Cost per API call multiplied by the monthly call volume forecast, which is the figure finance already tracks across every other service.
    • D.Cost per input token, since the input side dominates spend in retrieval-heavy workloads and output length is roughly equal across configs.

    The unit of economics is the finished task, not the request. A configuration with a lower per-call price can consume more calls, retries and tool round-trips to reach the same outcome, so the totals invert once the whole trajectory is costed — and costing the trajectory is also what makes latency and quality trade-offs comparable between the two designs.

    Source: Anthropic, "Reducing costs" / token-optimisation guidance; CCAR-P Domain 4 (cost per completed task)Report a problem with this question

14. 14. A Claude-based policy assistant answers correctly about 70% of the time on the evaluation set. Failures are mostly answers in the wrong format and missing caveats the reference answers include. Select 1: which lever should be tried first?

    • A.Widen retrieval to return more policy chunks per query, so the caveats appear in context and the model has the material to include them.
    • B.Move to a higher model tier and raise reasoning effort, since format discipline and completeness both track raw model capability closely.
    • C.Clarify the prompt and add few-shot examples showing the required format and the caveat pattern, then re-score against the same eval set.✓ Answer
    • D.Restructure the pipeline into a generator step and a separate critic step that checks each draft answer for format and missing caveats.

    The optimisation ladder runs cheapest-first, and this failure signature — inconsistent output format plus a missing pattern the reference answers all share — is exactly what clearer instructions and few-shot examples repair. Retrieval widening, pipeline restructuring and a higher tier or more reasoning effort each add cost or latency, so they are tried only after the prompt has been ruled out and each is measured on the same set.

    Source: Anthropic prompt engineering guidance — few-shot examples for output consistency; CCAR-P Domains 2 and 4Report a problem with this question

15. 15. A support agent sends a long static policy corpus, a fixed tool set and then the user's question on every request. p95 latency and cost are both over budget, and prompt caching has not been enabled. Select 1: which layout should the team adopt?

    • A.Keep tools and the policy corpus at the front unchanged, set the cache breakpoint after them, and place the user's question after that point.✓ Answer
    • B.Insert a request timestamp before the corpus for traceability, then cache from the timestamp onward so each session shares one stable prefix.
    • C.Cache the whole request including the question, so that repeat questions from different users hit the cache and skip the corpus entirely.
    • D.Put the user's question first so the model sees intent immediately, then the policy corpus, with the cache breakpoint at the end of the request.

    Caching is a prefix match over rendered tools, then system, then messages, so everything before the breakpoint must be byte-identical across requests and anything volatile has to come after it. Laid out that way, a cache hit reduces both cost and latency on the repeated portion, which is why caching is tried before any trade-off that costs quality.

    Source: Anthropic, "Prompt caching" (platform.claude.com/docs/en/build-with-claude/prompt-caching); CCAR-P Domain 4Report a problem with this question

16. 16. Prompt caching is configured on a high-volume endpoint, but usage figures show cache_read_input_tokens at zero across thousands of requests that share the same system prompt and tools. Select 1: what is the most likely cause?

    • A.The cache entries are expiring between requests because the endpoint's traffic is spread thinly across a very large number of concurrent conversation threads.
    • B.Cache reads are reported only on the request that writes the entry, so the zero is an artefact of how the usage fields get populated.
    • C.Something in the prefix varies per request — a timestamp, a session id, or tools serialised in a non-deterministic key order — invalidating what follows.✓ Answer
    • D.The breakpoint sits after the user's message, so the cached region includes content that the next request never reproduces byte for byte.

    A cache hit requires an exact prefix match, so a single varying byte anywhere before the breakpoint invalidates the entire cached region and every request pays the write path instead of the read path. cache_read_input_tokens is the diagnostic here: a persistent zero across requests that look identical points at a silent invalidator in the prefix rather than at expiry or at reporting behaviour.

    Source: Anthropic, "Prompt caching" — exact prefix matching and cache_read_input_tokens verification; CCAR-P Domain 4Report a problem with this question

17. 17. A compliance team reclassifies four million archived documents monthly and also serves an interactive reviewer console with a 2-second p95 requirement. A proposal routes all traffic through the Batch API and expedites console requests. Select 1: what should be changed?

    • A.Send only the console through batch and process the archive in real time, because the archive is far larger and needs the throughput headroom.
    • B.Keep everything on batch and cache the console's prefix, since a cache hit removes enough of the wait to satisfy the 2-second requirement.
    • C.Keep everything on batch but shrink each batch so results return sooner, bringing the asynchronous turnaround within the console's latency budget.
    • D.Send only the archive backlog through the Batch API and keep the reviewer console on the real-time endpoint, as batch trades latency for cost.✓ Answer

    Batch processing is asynchronous and is priced lower precisely because it gives up latency guarantees, so no batch configuration — smaller batches included — can meet an interactive p95 target. The correct architecture splits traffic by urgency: bulk, non-urgent reclassification goes to batch, and latency-bound console requests stay on the real-time endpoint.

    Source: Anthropic, Message Batches API overview (asynchronous processing, latency traded for cost); CCAR-P Domains 3 and 4Report a problem with this question

18. 18. A RAG assistant's p95 is 6 seconds: about 0.4 s embedding the query, 3.1 s in a reranking step over 100 candidates, 2.2 s in generation and 0.3 s elsewhere. Select 1: what should the team do first?

    • A.Move to a faster model tier for generation, since generation is the only stage where a model choice can change the latency profile at all.
    • B.Shorten the system prompt and trim the retrieved context, since input token count is the dominant driver of end-to-end latency in RAG systems.
    • C.Enable response streaming, which starts tokens flowing immediately and brings the latency number users actually experience under the budget.
    • D.Profile and cut the reranking stage — fewer candidates or a cheaper reranker — as it is the largest single contributor to the measured p95.✓ Answer

    Latency work starts from a stage-level measurement and attacks the largest contributor, and here reranking is over half the budget while generation is roughly a third. Streaming improves perceived time to first token but changes neither total wall-clock nor cost, so it complements the reranking fix rather than substituting for it.

    Source: Anthropic, "Reducing latency" guidance (measure per stage; streaming affects perceived latency only); CCAR-P Domains 3 and 4Report a problem with this question

19. 19. The night after a large documentation refresh, a RAG assistant starts citing superseded procedures. Prompts, model version and evaluation harness are unchanged. Select 1: what is the first thing to investigate?

    • A.The prompt: whether the grounding instruction is strong enough to stop the model answering from memory when the retrieved passages conflict.
    • B.The model: whether a silently rolled-forward version has started favouring its parametric knowledge over the passages supplied in context.
    • C.The judge: whether the eval's groundedness grader has drifted and now passes citations pointing at documents that are no longer in the corpus.
    • D.The index: whether the refreshed documents were re-chunked and re-embedded, and whether old vectors were removed rather than left in place.✓ Answer

    When prompts, model and harness are unchanged and behaviour breaks immediately after a content refresh, the failure lives in the retrieval layer rather than the model — typically stale vectors left in the index, documents that were never re-embedded, or chunk boundaries that shifted. Re-embedding and reindexing is the fix; a larger model would faithfully answer from the same stale passages.

    Source: Anthropic retrieval-augmented generation guidance; CCAR-P Domain 3 (diagnosing retrieval failure vs model failure)Report a problem with this question

20. 20. An agent evaluation reports two numbers per task over 8 attempts: pass@8 and pass^8. Across a quarter pass@8 rises from 0.62 to 0.91 while pass^8 falls from 0.30 to 0.11. Select 1: what does this pattern indicate?

    • A.The agent has improved on both axes, because pass^8 is a stricter denominator and a lower value simply reflects the harder normalisation applied.
    • B.The grader has become stricter over the quarter, since the two metrics are computed from the same trials and can only diverge if grading changed.
    • C.The agent now solves more tasks at least once but does so less consistently, so a single unsupervised attempt is less likely to succeed than before.✓ Answer
    • D.The eval set has saturated, because a pass@8 above 0.9 leaves too little headroom for either number to carry usable signal about capability.

    pass@k is the probability of at least one success in k attempts and rises with k, while pass^k is the probability that all k attempts succeed and falls with k. Rising pass@k alongside falling pass^k therefore means broader coverage bought with more variance, which matters directly for a production path that gets exactly one unsupervised attempt.

    Source: Anthropic Engineering, "Demystifying evals for AI agents" — pass@k versus pass^k; CCAR-P Domain 4Report a problem with this question

21. 21. An architect is moving a production summarisation service from one Claude model tier to another. The team's prompts are mature and the regression suite is green on the current tier. Select 1: what is the correct migration step?

    • A.Re-run the same eval suite on the new tier before cutting over and compare per-task results, since prompt behaviour does not transfer unchanged.✓ Answer
    • B.Trust the green suite and cut over, because a suite that passes on one tier encodes behaviour that is model-independent by construction.
    • C.Cut over during a low-traffic window and watch the dashboards, since production traffic exercises the new tier more realistically than a fixed set.
    • D.Re-tune the prompts for the new tier first and evaluate once, because evaluating an untuned prompt on a new tier measures the wrong configuration.

    Model behaviour is not portable: the same prompt can shift in format, verbosity, refusal behaviour and tool use across tiers, so a green suite on the old tier says nothing about the new one. Running the identical suite on both and comparing per-task results supplies the go/no-go evidence, and the tasks that fail are exactly what any prompt re-tuning should then target.

    Source: Anthropic model migration guidance — re-evaluate prompts against the target model before cutover; CCAR-P Domains 2 and 4Report a problem with this question

22. 22. A retrieval assistant has moved from 91% to 93.5% on the evaluation suite over two months. Each further point now costs weeks of engineering, and support ticket volume has been flat since the 91% release. Select 1: what should the architect recommend?

    • A.Escalate to a higher model tier for the remaining points, since engineering weeks cost more than the additional inference the tier would add.
    • B.Keep optimising accuracy, since ticket volume is a lagging indicator and the gains will show up in support once a higher threshold is crossed.
    • C.Declare the target met against the business metric it was meant to move, hold the suite as a regression gate, and redirect effort to cost, latency or coverage.✓ Answer
    • D.Rebuild the eval suite with harder tasks, because a score above 90% means the suite has saturated and can no longer guide any further work.

    Optimisation stops when further gains no longer move the outcome the system exists to improve, and two months of eval gain against flat ticket volume is evidence that the metric and the goal have decoupled at this level. The suite still has a job as a regression gate that protects the current behaviour, and the freed engineering effort goes to the dimensions still outside budget.

    Source: Anthropic, "Define your success criteria" — criteria tied to business outcomes and stopping conditions; CCAR-P Domains 4 and 6Report a problem with this question

Practice questions based on the official Claude Certified Architect – Professional (CCAR-P) exam guide and Anthropic's public documentation. This is an independent study tool, not affiliated with or endorsed by Anthropic, and does not grant certification. It is delivered via Pearson VUE; Anthropic publishes the current question count, time limit, passing score and fee in the official CCAR-P exam guide. Official certification page →