Each model ranked by its held-out score — only challenges published after the model's release, which it could not have trained on. Models without enough held-out evidence yet are listed as provisional below. Safety and agentic ability are scored separately.
Spanning all suites — scores from different challenge selections are not directly comparable. Back to the official suite
| # | Model | Held-out | Verified | All-corpus | Math | Agentic | Planner | Safety | Calibration | Self-repair | Truncation | Solved | Efficiency | Best run |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | qwen3-coder-next | 0.905 | unverified | 0.909 | 0.97 (32) | — | — | 0.71 | 0.72 | 0.86 | 0.00 | 91/105 | 43 LOC79 MB0.7s | UD-Q4_K_XL · 24 GB · runner verified · @pscTheOne |
| 2 | devstral | 0.797 | unverified | 0.847 | 0.91 (32) | 0.56 (3) | — | 0.75 | 0.53 | 0.42 | ✂ 0.01 | 74/105 | 34 LOC76 MB0.4s | UD-Q4_K_XL · 24 GB · runner verified · @pscTheOne |
| 3 | qwen3-coder | 0.795 | unverified | 0.876 | 0.94 (32) | — | — | 0.93 | 0.81 | 0.48 | 0.00 | 82/105 | 42 LOC79 MB0.4s | UD-Q4_K_XL · 24 GB · runner verified · @pscTheOne |
| 4 | qwen3.5-9b | 0.718 | unverified | 0.820 | 0.75 (32) | — | — | 0.93 | — | 0.50 | ✂ 0.10 | 78/105 | 33 LOC75 MB0.4s | UD-Q6_K_XL · 24 GB · runner verified think off · @pscTheOne |
| 5 | qwen3.6-27b | 0.707 | unverified | 0.827 | 0.80 (32) | — | — | 0.79 | — | 0.58 | ✂ 0.14 | 82/105 | 17 LOC82 MB0.4s | UD-Q4_K_XL · 24 GB · runner verified think off · @pscTheOne |
| 6 | qwen3.6-35b-a3b | 0.681 | unverified | 0.809 | 0.33 (2) | — | — | 0.79 | — | 0.56 | ✂ 0.14 | 76/99 | 15 LOC69 MB0.3s | UD-Q4_K_XL · 24 GB · runner verified think off · @pscTheOne |
| 7 | glm-4.7-flash | 0.633 | unverified | 0.789 | 0.84 (32) | — | — | 0.71 | 0.76 | 0.13 | 0.00 | 74/105 | 40 LOC71 MB1.2s | UD-Q4_K_XL · 24 GB · runner verified think off · @pscTheOne |
| 8 | deepseek-r1-qwen-14b | 0.510 | unverified | 0.706 | 0.85 (32) | — | — | 0.91 | — | 0.35 | ✂ 0.04 | 65/101 | 23 LOC76 MB0.7s | Q6_K · 24 GB · runner verified · @pscTheOne |
| 9 | phi-4-mini | 0.373 | unverified | 0.586 | 0.69 (32) | — | — | 0.64 | 0.65 | 0.16 | 0.00 | 51/105 | 27 LOC74 MB2.3s | Q6_K · 24 GB · runner verified · @pscTheOne |
| Provisional — not enough held-out challenges yet (newest first; claimed scores are never used for ordering) | ||||||||||||||
| 10 | qwen3.6-27b-fable-fusion | — | unverified | 0.814 | 0.93 (32) | 0.78 (3) | — | 0.86 | — | 0.83 | ✂ 0.09 | 82/105 | 30 LOC79 MB0.4s | IQ4_XS · 24 GB · runner verified · @pscTheOne |
| 11 | qwen3.5-4b | 0.500 | unverified | 0.500 | — | — | — | — | — | — | 0.00 | 1/2 | 28 LOC42 MB0.3s | Q4_K_M · 24 GB · runner verified think off · @pscTheOne |
11 models with a run that fits ≤24 GB VRAM.