Claude Code Effort Levels: 45 Runs Measuring What Actually Changes

Search claude code effort levels and the first page agrees with itself. Low is for mechanical work, max is for the hard stuff, and higher effort buys you a better answer. The official docs describe the dial; the write-ups around them rank the tiers by how smart each one is.

None of them run the same task five times and show you what came back.

So I did. Three tasks, five effort levels, three trials each — 45 sessions of claude -p, all on Opus 5, scored against answers computed by actually executing the code. The answers barely moved. Everything else did.

The rig#

claude takes the dial as a flag, which makes this measurable without touching settings:

claude -p "$PROMPT" --effort low --output-format stream-json --verbose

Every run is a fresh session; nothing carries over. The numbers come from the result event in the JSON stream — duration_ms, num_turns, usage.output_tokens, and usage.output_tokens_details.thinking_tokens, which is where the dial actually shows up.

Three tasks, sitting at different depths:

• T1 — turn a 17-row CSV into a Markdown table. Deterministic. One right answer, no reasoning to do.
• T2 — 200 exported functions across 41 files. Name every one that throws when you pass it "".
• T3 — the same question against a codebase built to punish skimming.

T2 cannot be answered with grep. The dangerous code also lives in helpers.ts, so a call site can look harmless; guards appear at random, and only some of them work:

export function mod004Handler2(input: string): string {
  if (!input) return "none";         // "" is caught. Safe.
  return parseTag(input);
}

export function mod017Service2(input: string): string {
  if (input === null) return "none"; // "" is NOT null. Falls straight through.
  return pickId(input);              // "".match(/id=(\d+)/) is null -> null[1] throws
}

T3 goes further: the helpers form a three-level chain, and the argument gets rewritten on the way down.

// helpers-b.ts
export function decorateTag(raw: string): string {
  return takeSecond(raw + ",fallback");  // "" becomes ",fallback" -> does NOT throw
}
export function normalizeTag(raw: string): string {
  return takeSecond(raw.trim());         // "" stays "" -> takeSecond throws
}

Two call sites that look identical resolve to opposite answers three frames down. Of the 200 functions, 86 sit at depth 3, 78 at depth 2, and 167 have their argument transformed somewhere along the way.

I did not trust my own answer key. A verify script strips the type annotations, loads every generated file into Node, calls all 200 functions with "", and records which ones actually throw. The generator’s claims and the runtime’s behaviour agree on all 400 functions across both codebases. A second script checks every run’s tool calls for a peek at the answer files or at previous session logs in ~/.claude/projects/ — 45 runs, zero hits.

What came back#

T1 — CSV to Markdown table (deterministic)

effort	n	sec	tool calls	thinking tok	output tok	correct
low	3	7.3	1.0	0	498	100%
medium	3	7.0	1.0	0	498	100%
high	3	6.5	1.0	0	498	100%
xhigh	3	11.8	2.0	0	594	100%
max	3	9.1	1.7	0	561	100%

T2 — 200 functions, one level of indirection

effort	n	sec	tool calls	thinking tok	output tok	recall
low	3	44.2	5.7	2,088	3,148	99.5%
medium	3	80.5	20.7	3,931	6,549	100%
high	3	107.1	45.3	5,795	10,052	100%
xhigh	3	107.1	44.7	6,238	10,316	100%
max	3	136.9	46.0	10,007	14,372	100%

T3 — same question, three-level helper chain

effort	n	sec	tool calls	thinking tok	output tok	recall
low	3	80.3	7.3	5,602	7,062	100%
medium	3	120.3	22.7	8,312	11,245	100%
high	3	147.4	23.7	11,487	14,430	100%
xhigh	3	187.8	35.3	14,730	18,497	100%
max	3	199.7	46.0	16,246	20,691	100%

Precision was 100% everywhere: not one false positive in 45 runs, across 1,995 opportunities to name a function that does not throw. The single error in the whole experiment is one missed function in one low run of T2.

Three things that fell out#

Effort does nothing when there is nothing to think about. T1 reports zero thinking tokens at all five levels. Low, medium and high are identical down to the output token — 498, three trials each. Only xhigh and max spend a second Read checking their own work, and it changes nothing.

What the dial actually moves is the search strategy. Low reads T2 in one shot:

Bash: cat mod0*.ts mod1*.ts

High opens the same files one at a time, then re-prints them with an awk separator to inspect them again — 45 tool calls against low’s 5.7. It is more thorough in a way you can watch. On this task the thoroughness buys nothing, because low already had the answer.

The one miss is the kind you would expect. Here is the function low dropped:

export function mod017Service2(input: string): string {
  if (input === null) return "none";  // "" is not null — falls through
  return pickId(input);               // throws
}

It saw a guard and took it at face value. That is the failure mode extra reasoning is supposed to catch, and one step up the dial did catch it — medium and above got this function right in all twelve runs. So the dial is not inert. It is just that on this shape of problem, the gap it closes is 1 function in 1,995.

What I got wrong#

I built T2 expecting to find the ceiling, watched low score 100%, and assumed the task was too easy. So I built T3 — three-level chains, arguments rewritten mid-flight, guards that only work half the time — and low scored 100% on that too, three times out of three.

Two attempts to construct a task where effort matters, two failures. Reachability analysis over 200 functions is apparently not, for Opus 5, a problem that needs more thinking; it needs the files read. Once they are read, low already knows the answer, and the extra 11,000 thinking tokens max spends have nothing left to find.

The other thing I expected was a clean ladder. xhigh does not sit between high and max: on T2 it ties with high to the tenth of a second (107.1 vs 107.1), and on T3 it lands next to max. Whatever separates those two tiers did not show up in any of my measurements.

Limits#

• Every task here has one correct answer. Design decisions, refactoring strategy, anything where the work is choosing between defensible options — untested, and that is exactly where the dial has room to matter.
• Opus 5 only. The ceiling almost certainly sits somewhere else on Sonnet or Haiku.
• Three trials each. The 99.5% on T2 low is one miss in three runs, so treat the frequency as a hint, not a rate.
• Single-turn claude -p sessions. Long interactive work may behave differently.
• I stopped reporting cost. It swings about 30% with prompt cache hits between otherwise identical runs, which makes it useless for comparing tiers. Token counts are the honest measure.

What I do now#

Leave it at the default and stop thinking about it. On work with a verifiable answer, the difference between low and max in this experiment was one function out of 1,995, bought with 2.5–3x the wall-clock and up to 4.8x the thinking tokens.

The case for raising it is not “the answer will be better.” It is that low occasionally takes a guard at face value, and one step up fixed that every time here. If you are asking about a codebase you cannot check by hand, medium is cheap insurance. Past medium, on this kind of question, you are paying for a longer wait and watching the model read the same files twice.

I would rather be shown wrong on the open half of this: if you have a task where max reliably beats medium, that is the interesting experiment, and it is not this one.