What he said
Anthropic added eval tooling to Claude Code. An eval is a repeatable test that checks whether your AI feature behaves the way you want. The new build_eval and hill-climb commands in the claude-api plugin help you build those tests, check the graders that score them, and improve your app against them.
Hamel Husain tried it on conversation logs from an apartment leasing assistant. His verdict: “I’d hold off for now.” His main objection is about order. In his words, “I believe you should be looking at data first to inform your understanding and prioritize which evals to write.”
Who he is and why it carries weight
Husain is an independent machine learning engineer who focuses on evals. His site says he is “helping teams debug, analyze, and measure their systems,” and that he co-teaches an evals course for engineers and product managers. He also says he usually doesn’t review eval tools, because they change too fast. A first-party tool from Anthropic, he argues, is likely to shape how many people approach evals, which is why he made an exception.
What he found
Husain describes three problems. First, Claude proposed several possible failures and asked him to pick one right away. He and his co-host hadn’t read the conversations yet, so they couldn’t tell whether the recommended failure (call-transfer rules) was real or worth prioritizing. They went along with it because they figured that is what typical users would do.
Second, the tool asked them to check its labels by skimming a Markdown file and telling Claude in chat which labels were wrong. His annotated screenshot calls that “the most painful way you can think of to label data” and suggests building a small web app for it instead.
Third, the grader bundled four checks into a single pass or fail result: confirm once, transfer immediately, stay silent, and avoid saying tool mechanics aloud. Husain prefers testing one error at a time, or at least separating code-based checks from judgments made by an AI model.
He praised the tool’s ability to discover issues in the logs. He also notes that Anthropic’s introductory guide agrees on reading data, sampling examples, and avoiding tests that the app already passes easily. His criticism concerns the workflow he observed in this test.
Read logs before deciding what to test
Read real outputs before deciding which failures deserve tests. A test selected without that review can measure a failure the tool guessed was important. It may not measure what is actually hurting your users. A combined score also needs diagnostics. When one pass or fail covers four behaviors, the score alone doesn’t tell you which behavior broke.
The critique has limits. This is one livestream on one dataset, a leasing assistant, so it shows how the tool behaved there, not how it behaves everywhere. Husain also expects the plugin’s developers to update it, so these may be early rough edges rather than design choices. His headline rule, that a tool is “not worth using” if it doesn’t put looking at data at the center of your workflow, is a strong statement of his own method, not a tested comparison of eval tools.
There is also a middle path he points toward. The tool’s issue discovery is useful as a source of hypotheses. The risk is treating those hypotheses as a finished to-do list.
What builders should do
Use the tool’s issue discovery to find examples worth reading. Choose tests after checking which failures actually occur and what they cost users.
- Read a sample of real conversations or outputs yourself before accepting any proposed eval. Write down the failures you actually see.
- Pick the failure that costs you the most, then ask for an eval on that one failure only.
- Keep results for each check, or split the grader where necessary, so you can identify the cause of a failure.
- Check the grader’s own labels against examples you have read. A grader you haven’t checked can look confident and still be wrong.
End of article