Published full-model reference, then measured local recipes.
| Recipe | GPQA | Terminal | LCB | IFBench | Auto |
|---|---|---|---|---|---|
| BF16 / publishedPublished reference | 89.2 | 73.0 | 90.3 | 79.5 | — |
| 73.2 | 30.3 | 66.0 | 42.3 | 18.3 |
The published full-model reference is the top row — or the dashed gold tick on Graphic — not rank #1. A blank cell, missing tick, or missing bead has no recorded cite. Scores are never invented.
ON THIS SETUP · MEASURED BY LOCALSONLY
Local GB10 · 128GB · llama.cpp
Solve programming problems with executable tests.
Version 6.0 · 100 tasks
Generated solutions are checked by executable tests. The benchmark version and problem window matter; this is different from maintaining an existing repository.
Multi-step tasks in a terminal environment.
Version 2.1 · 89 tasks
The task must pass its verification in a terminal environment. Tools, harness, context budget and execution failures all contribute to the result.
Follow verifiable instruction constraints.
Source-reported protocol · 300 tasks
Read the source receipt for the exact grading rule and evaluation settings. A reported aggregate does not identify which individual tasks passed or failed.
Sales, marketing, operations, support, finance and HR workflows.
Local evaluation · 300 tasks
A workflow is complete only when every assertion passes. Mean reward gives partial credit for progress; it is not the share of jobs finished. Task sets and evaluation budgets can differ between receipts.
Mean reward 48.8% · partial credit, not completed jobs.
Graduate-level science questions.
Diamond · version 1.0 · 198 tasks
Credit comes from answering the science question correctly. Multiple-choice accuracy does not establish research or experimental competence.
BEYOND THE HEADLINE
Each block represents one recorded task, in receipt order from left to right, then top to bottom. Fill shows the outcome; work-category colors are not used here.
Tested by LocalsOnly · UD-Q4_K_XL · GB10 · 128GB. Partial progress still requires further work before a task is complete.
Inspect individual task outcomes →Completed the recorded checks.
4 / 4 assertions · 13 steps
Follow this exact task ↗sales.update_contact_phone · ID 1
sales · 4 / 4 assertions · 13 steps / 24 tools · passed
The attempt stopped before completion. The record does not establish why.
0 / 2 assertions · 26 steps
Follow this exact task ↗sales.create_important_draft · ID 9
sales · 0 / 2 assertions · 26 steps / 98 tools · aborted_rollout
Context, sampling, harness settings and dataset versions can vary by benchmark. Each evidence link above contains its run-specific configuration.