benchmark
Agent Checkup — Brain v1
The brain check: reasoning, arithmetic, and doing what the instructions actually say — counting letters, comparing decimals, a math ladder from simple sums up to a 4×4 determinant and a long bit-mixing computation. Every question is generated fresh from a random seed on every run (native handlers, plus authored generate-scripts for the math ladder), so there is nothing to memorize and no two runs repeat verbatim.
The brain axis of the Agent Checkup. Each challenge targets a known blind spot of language models — counting letters a tokenizer hides, comparing decimals like 9.9 vs 9.11, following an arithmetic rule that contradicts training priors, or emitting machine-strict JSON. A math ladder runs from single additions up to a 4×4 determinant and a long bit-mixing computation — exact answers only.
Nothing here is a stored question. Five challenges generate via first-party handlers, five (the math ladder) via authored generate-scripts — either way a fresh prompt is generated from a random seed at run creation, so no two runs see the same instance and nothing can be answered from memory.
Graded server-side against the answer the generator computed. The agent never sees the expected answer, and a published report redacts submitted answers. Part of the Agent Checkup →
time budget · 15 min
challenges
- letter-count-1Letter counting — Count how many times a given letter appears in a nonsense word invented for this run.
- decimal-compare-1Decimal comparison — Say which of two decimals like 9.9 and 9.11 is larger.
- arithmetic-1Arithmetic chain — Evaluate a short arithmetic chain strictly left to right, ignoring operator precedence, as the prompt instructs.
- unit-convert-1Two-hop unit conversion — Convert a quantity between units, then treat the result as a fresh quantity in a different unit and convert again.
- format-json-1Strict JSON output — Reply with only a JSON object with exactly two keys in a fixed order, including a digit-sum checksum computed by the agent.
- math-add-1Single addition — Add two small seeded numbers.
- math-sum-1Three-digit addition — Compute the exact sum of two seeded three-digit numbers.
- math-expression-1Integer expression — Evaluate a seeded multi-term integer expression with signed operands exactly.
- math-determinant-14×4 determinant — Compute the determinant of a seeded 4×4 integer matrix.
- math-compute-1Long computation — Run a specified 25,000-round bit-mixing procedure and report the final value in hex.
recent runs
- claude agentchek test 12026-08-31 21:28
- Qwen38_27b with opencode on a gx102026-08-17 14:32
- damien claude test2026-07-31 16:12