Quote boards, one exam at a time

Honest ranking is the same exam. These tables are WorkChi pins of public LiveBench items, scored with official generators and official scorers. They are not livebench.ai Overall, not Artificial Analysis Intelligence, and the coding table is not site Coding 83.9.

Snapshot 2026-08-31 · methodology · business-task leaderboard

Isolated protocol · n=48

HS-48

Same 48-item LiveBench HS pin, official-gen yaml/floor applied (temp 1, ≥32k; yaml where shipped, floor otherwise). Default effort — not Max/xHigh.
Headline is item-pooled pass rate. Language / reasoning / data analysis are category slices, not a second ranking.
Not livebench.ai Overall. Ties at n=48 are common — read the tier, and use mathif to separate the cheap-model cluster.
Read `tier`, not the row order: rows sharing a tier are not separable on this pin (exact paired sign test over the shared items, alpha 0.05). At n=48 one standard error is ~7pp.
`ci95` is Wilson on the pooled pass rate; `runCount` is how many replicate seats were pooled and `spreadPp` their max-min. Repeat seats pool item-wise — the board never publishes the best of k.
A second t0/4k block exists on disk. It is not this exam and is not shown here.
A model id without a date or version is a floating alias and can move under you: `deepseek/deepseek-v4-flash` and the pinned `deepseek/deepseek-v4-flash-0731` sit in different tiers on identical items, 60.4% against 70.8%. Prefer the pinned id when quoting a row.
Rows that lost more than 10% of their cells to the transport (DNS, dropped sockets, provider timeouts) are off this board — a lost cell scores zero, so they understate the model. A model's own empty or truncated answers still count against it.

Tier	#	Model	Pass %	Pass	Language	Reasoning	Data	p50	Cost
T1	1	Claude Opus 5 anthropic/claude-opus-5	77.1	37/48 63–87%	81.3	93.8	56.3	9819ms	$1.52
T1	=	Claude Sonnet 5 anthropic/claude-sonnet-5	77.1	37/48 63–87%	81.3	93.8	56.3	1890ms	$1.44
T1	=	Gemini 3.7 Flash google/gemini-3.7-flash	77.1	37/48 63–87%	81.3	93.8	56.3	2132ms	$0.26
T1	4	fusion:lightning fusion:lightning	75.0	36/48 61–85%	75.0	93.8	56.3	2988ms	$0.35
T1	5	Grok 4.6 x-ai/grok-4.6	72.9	35/48 59–83%	81.3	93.8	43.8	39.8s	$0.10
T1	6	deepseek-v4-flash-0731 deepseek/deepseek-v4-flash-0731	70.8	34/48 57–82%	75.0	93.8	43.8	569ms	$0.13
T1	=	Gemini 3.6 Flash google/gemini-3.6-flash	70.8	68/96 61–79% · k=2	78.1	93.8	40.6	1512ms	$2.17
T1	8	glm-5.3-flash @cf/zai-org/glm-5.3-flash	68.8	33/48 55–80%	68.8	87.5	50.0	27.9s	$0.06
T1	=	fusion:field fusion:field	68.8	33/48 55–80%	62.5	81.3	62.5	2105ms	$1.40
T1	=	Gemma 4 31B google/gemma-4-31b-it	68.8	33/48 55–80%	75.0	81.3	50.0	228ms	$0.02
T1	=	GPT-5.6 Sol openai/gpt-5.6-sol	68.8	33/48 55–80%	75.0	87.5	43.8	11.3s	$1.07
T1	=	GPT-5.6 Terra openai/gpt-5.6-terra	68.8	33/48 55–80%	81.3	87.5	37.5	7401ms	$0.26
T1	=	glm-5.3 z-ai/glm-5.3	68.8	33/48 55–80%	68.8	87.5	50.0	2182ms	$1.40
T1	14	MiniMax M3 minimax/minimax-m3	67.7	65/96 58–76% · k=2	71.9	87.5	43.8	22.9s	$0.58
T2	15	DeepSeek V4 Flash deepseek/deepseek-v4-flash	60.4	29/48 46–73%	75.0	56.3	50.0	347ms	$0.02
T2	=	GLM 5.2 z-ai/glm-5.2	60.4	29/48 46–73%	62.5	75.0	43.8	12.2s	$0.83
T2	17	Gemini 3.5 Flash-Lite google/gemini-3.5-flash-lite	52.1	25/48 38–66%	62.5	50.0	43.8	435ms	$0.12
T3	18	Ling 3.0 Flash inclusionai/ling-3.0-flash	27.1	13/48 17–41%	12.5	18.8	50.0	507ms	$0.03

Isolated protocol · n=32

Math + instruction following

32-item pin: mathematics + instruction following. Ranked by cat-mean (task-mean, then category-mean).
effort=high is not livebench.ai Max / xHigh. Not official LiveBench Overall.
Do not average this table with HS-48.
Rows that lost more than 10% of their cells to the transport (DNS, dropped sockets, provider timeouts) are off this board — a lost cell scores zero, so they understate the model. A model's own empty or truncated answers still count against it.
Read `tier`, not the row order. `ci95` is Wilson on pass rate; repeat seats of one serving stack are pooled item-wise, not best-of-k.

Tier	#	Model	Cat-mean	Pass	Math	IF	p50	Cost
T1	1	Gemini 3.7 Flash google/gemini-3.7-flash	93.0	160/182 82–92% · k=2	90.0	95.9	2241ms	$2.23
T1	2	GPT-5.6 Sol openai/gpt-5.6-sol	92.0	157/182 81–91% · k=2	90.3	93.7	614ms	$2.89
T1	3	Gemini 3.6 Flash google/gemini-3.6-flash	91.6	155/182 79–90% · k=2	90.9	92.4	1558ms	$4.04
T1	4	Claude Opus 5 anthropic/claude-opus-5	90.0	27/32 68–93%	88.1	91.9	1650ms	$1.27
T2	5	GPT-5.6 Terra openai/gpt-5.6-terra	88.9	145/182 73–85% · k=2	90.5	87.2	598ms	$2.20
T2	6	fusion:lightning fusion:lightning	86.7	25/32 61–89%	73.5	100.0	2479ms	$0.60
T2	7	fusion:balanced fusion:balanced	83.4	112/150 67–81%	81.5	85.4	17.3s	$3.33
T2	8	Gemini 3.5 Flash-Lite google/gemini-3.5-flash-lite	82.8	25/32 61–89%	65.5	100.0	1209ms	$0.61
T2	9	fusion:fast fusion:fast	81.9	110/150 66–80%	74.0	89.8	1619ms	$1.65
T2	10	MiniMax M2.7 minimax/minimax-m2.7	81.9	25/32 61–89%	67.6	96.1	42.7s	$0.27
T2	11	GLM 5.2 z-ai/glm-5.2	81.5	25/32 61–89%	75.3	87.8	472ms	$0.33
T3	12	MiniMax M3 minimax/minimax-m3	81.5	230/332 64–74% · k=3	86.7	76.3	12.6s	$2.17
T3	13	DeepSeek V4 Flash deepseek/deepseek-v4-flash	79.0	24/32 58–87%	62.2	95.8	1220ms	$0.05
T3	14	Gemma 4 31B google/gemma-4-31b-it	75.1	22/32 51–82%	66.6	83.6	422ms	$0.14
T3	15	Grok 4.6 x-ai/grok-4.6	74.8	24/32 58–87%	57.2	92.3	53.6s	$0.09
T3	16	Claude Sonnet 5 anthropic/claude-sonnet-5	74.6	22/32 51–82%	73.6	75.7	1934ms	$1.37
T3	17	Ling 3.0 Flash inclusionai/ling-3.0-flash	74.6	24/32 58–87%	49.1	100.0	600ms	$0.04
T3	18	fusion:field fusion:field	69.3	18/32 39–72%	58.2	80.4	60.4s	$2.26

What these numbers are

HS-48 is language + reasoning + data analysis (no math, no coding, no IF), default effort. Math+IF and coding-v2 use effort=high, which is still not livebench.ai Max / xHigh. Coding-v2 is a 2024 public slice (removal_date=2025-04-02); Opus and Gemini 3.6/3.7 all score 100 on it. Rank that pair on math+IF, not coding-v2.

Fusion rows are ensembles, not single models. Serving is mixed: MiniMax and xAI first-party when available, everyone else OpenRouter. Do not average a row with a different protocol string.

A separate residual coding pin (n=3, not shown as a ranking table) keeps the leftover items Opus and Gemini 3.7 both fail. On that exam Sol scores 1/3; Opus, Gemini 3.6, and Gemini 3.7 score 0/3. n is too small to quote a ranking. It is not coding-v2 and not site Coding.

Loading…

LiveBench subsets