Does 'think step by step' help AI count letters?

The question

Counting letters is a classic AI stumble. We gave three-word phrases and asked how often one letter appears. Does asking it to spell the words out and count step by step beat a straight answer?

What we found

• Haiku: 97% to 100% (no clear difference)
• Sonnet: 90% to 97% (no clear difference)

"Clear" means the two 95% ranges below do not overlap. When they overlap we say there is no clear difference, even if the percentages differ. With about 30 answers per cell, small gaps are noise.

Model	How we asked	Passed	Rate	95% range
Haiku	Asked for just the number	29 of 30	97%	83% to 99%
Haiku	Told to spell it out and count step by step	30 of 30	100%	89% to 100%
Sonnet	Asked for just the number	27 of 30	90%	74% to 97%
Sonnet	Told to spell it out and count step by step	29 of 30	97%	83% to 99%

What counts as a pass: the final number is correct.

What we saw reading the answers

• Both ways of asking were mostly right. The few misses were off by one on the longest phrases, for example answering 11 when the truth was 10.
• Step by step did not fix every miss. In one Sonnet answer the spelled-out list itself was miscounted, so showing the work does not guarantee a right answer.
• What to do: for counting that matters, use a program or a spreadsheet, not the AI's memory of the word.

The exact wording

Shown with the first question as the example.

Asked for just the number

System prompt: You are a helpful assistant.

How many times does the letter "r" appear in these words: "strawberry raspberry blueberry"? Reply with only the number.

Told to spell it out and count step by step

System prompt: You are a helpful assistant.

How many times does the letter "r" appear in these words: "strawberry raspberry blueberry"? Spell each word out one letter per line, count the matches step by step, then finish with a line "Answer: N".

How we tested

• Models: Haiku (claude-haiku-5-5); Sonnet (claude-sonnet-5-5). Claude models only. We have not tested ChatGPT, Gemini, Grok or others, and results can differ.
• 10 different questions, each asked 3 times per way of asking, so the answer's randomness is included.
• Every answer was scored by a short program, not by a person's opinion. We then read a sample by hand.
• Each question was sent on its own through the Claude command-line tool, with tools, memory, settings and conversation history switched off, and no web access or examples. That tool still adds a short fixed note about the machine (operating system, working folder, the account's email domain), about 900 tokens in all. It is identical for both ways of asking, so comparisons stay fair, but absolute numbers may differ a little from a plain API call. We redacted the owner's company name from one published answer where a model repeated it.
• Run on 2026-10-26. Model versions change; treat results as a snapshot.

Limits

Ten three-word phrases. Newer models may have been trained on this kind of puzzle.

Raw data

Every question, answer and score is in the raw data file (filter by "exp": "count-letters"). Re-run it yourself with the harness in the project's testlab folder, or paste the prompts above into your own account.