Agentic reasoning benchmarks, by tier
Each tier adds one source of information the prompt doesn’t contain. Click a row to see one item taken from that split’s data files: the task as stored, the tools, the reference the dataset keeps (gold calls, oracle actions or answer) and how it’s scored. Every split has an item from the data.
Tier 0 No new information: everything is in the prompt Tier 1 Information from the environment Tier 2 Information from the user Tier 3 Information from another agent
Evaluation
Answer
State
Action match
LLM judge
Decision
Tier Benchmark Split Units Hops Evaluation Tags
Hops count dependent calls, where one call’s output is the next call’s input. “Parallel” means several independent calls. Units counts the items behind each row (tasks, questions, scenarios or rounds); hover or read the small print for how they relate to the benchmark total. Tags are free-form labels for what each row tests and can be searched. A tag under the tier (e.g. 0↔1, 1↔2, 2↔3) marks a split whose information source sits between two tiers or mixes them.
Evaluation families: answer (forced-answer curves work directly), state (needs rollouts with state restore at each cut), action match (fits commitment on which tool and which arguments), LLM judge (judge noise adds up across cuts), decision (only the call / ask / refuse choice is scored).
Sources