Mix of X instead of Best of X

Many benchmarks generate multiple outputs per question and mark the result as a pass if any one output is correct ("best of X"). Here, scores are averaged across all runs. For example, if a question is worth 5 points and four runs score 5, 0, 0, and 4, the final score for that question is 9/4 = 2.25.

Two evaluation methods

Questions are evaluated either by a judge LLM or by a custom verifier script. The judge LLM (Gemini 3.0 Flash in my case) has access to the ground truth and marks answers as pass or fail. Verifier scripts are written specifically for individual questions and programmatically check the model's output.

Partial credit

Some questions support partial points, but only when evaluated by verifier scripts. I don't rely on judge LLMs for partial scoring. With script-based verification, partial credit has been reliable.

Token limits tied to question value

Each question has a point value, and the maximum token limit scales with it. A 1-point question uses a base limit of 8,196 tokens, while a 5-point question allows up to roughly 40k tokens. Harder questions are given more room for reasoning.

Gradual release of questions

The repository is open source, but the full question set is not publicly available yet. This is to avoid future models training directly on the benchmark. Instead, I will release questions worth about 10% of the total points each month when I run new evaluations and replace them with new questions. The first batch is already published on the website.

Dynamic point adjustment

After initial runs, I noticed that some questions were misweighted. To reduce personal bias, I introduced an automatic adjustment system. If all models fully solve a question, its point value is reduced. If none succeed, the value increases. Intermediate outcomes are adjusted proportionally. A secondary leaderboard based on this dynamic scoring is also available.

Controlled model and provider selection

OpenRouter models are used with at least FP8 quantization for open-source models. Providers were selected based on accumulated community feedback and broader observations. Certain providers were excluded due to consistently poor API performance.

Varied and original questions

• Basic Mix: Simple tasks or altered well-known questions to test overfitting.
• General Knowledge: Deep knowledge checks and "future prediction" questions (events that happened after model cutoff).
• Math: Medium to hard problems sourced from private sources.
• Reasoning: Logic and puzzle-based questions, including chess and word puzzles.

Broad model coverage

The benchmark includes leading proprietary models, strong open-source options, and models that can realistically run on consumer GPUs. I'm open to suggestions for missing models.

High reasoning effort

All requests are sent with reasoning effort set to high, where supported by the model.