ARC-AGI-2
A hard abstract-reasoning benchmark from ARC Prize focused on novel task generalization.
Benchmarks
Human-readable benchmark definitions with score direction, limitations, and source records.
Humanity's Last Exam (HLE)
A broad expert-level benchmark intended to test difficult questions across many domains and formats.
FrontierMath
An advanced mathematics benchmark curated around difficult research-style math problems.
GPQA / GPQA Diamond
A graduate-level, Google-proof question-answering benchmark with a commonly used Diamond subset.
MMLU-Pro
A harder MMLU-style benchmark for broad knowledge and reasoning.
SWE-bench Verified
A curated SWE-bench split for resolving real software issues with agentic coding systems.
SWE-bench Pro
A professional software-engineering agent benchmark with public and private leaderboard splits.
Terminal-Bench 2.x
A terminal-environment agent benchmark for command-line task completion.
MLE-bench
A machine-learning engineering benchmark based on Kaggle-style competition tasks.
BrowseComp
A browsing benchmark for hard-to-find information and deep web research tasks.
tau-bench / tau2-bench
A tool-use benchmark family for business-process and customer-service style agent tasks.
OSWorld / OSWorld-Verified
A computer-use benchmark for operating-system and GUI agents.
WebArena
A web-agent benchmark for completing tasks in realistic self-hosted web environments.
VisualWebArena
A visual web-agent benchmark extending WebArena with image-grounded web tasks.
LongBench v2
A long-context understanding benchmark with a project leaderboard.
RULER
A synthetic long-context stress benchmark from NVIDIA.
HELMET
A long-context benchmark suite focused on practical application-style tasks.
MMMU-Pro
A harder multimodal academic benchmark derived from MMMU-style tasks.
Video-MME
A video understanding benchmark for multimodal models.
MathVista
A visual math reasoning benchmark for multimodal models.
CharXiv
A chart and figure understanding benchmark for scientific visual reasoning.
SimpleQA Verified
A factuality benchmark for short-answer factual knowledge questions.
FRAMES
A RAG and factual multi-hop question answering benchmark.
GraphWalks
A graph reasoning benchmark candidate that still needs a stable authoritative source before launch.
TerminalWorld-Verified
A terminal-agent benchmark candidate that needs source confirmation.