Does "let's think step by step" actually work?

It does measurably improve accuracy on multi-step problems with older non-reasoning models, which is why the phrase became well known. Naming the specific steps you want works better, and with reasoning models the instruction is redundant.

Does chain-of-thought prompting cost more?

Yes. The reasoning is generated output, so you pay for those tokens and wait for them. That is precisely the trade: more computation spent on the problem, in exchange for money and latency. It is worth it on hard problems and wasteful on easy ones.

Can chain-of-thought make the answer worse?

On short creative or stylistic tasks, yes. The deliberation shows up in the prose as over-explanation and hedging. It can also let a model talk itself out of a correct first instinct on simple factual questions.

Is the reasoning a model shows its real reasoning?

Not necessarily. Generated reasoning is text produced by the same process as any other output, and it can be a plausible narrative rather than a faithful account. It is still useful — both because generating it improves accuracy and because it exposes checkable intermediate claims — but it should be read as a hypothesis, not a transcript.

Should I use chain-of-thought with reasoning models?

Usually not. They already deliberate internally, so the instruction is redundant and can interfere. Spend the space on a precise goal and clear constraints instead.

Chain-of-thought prompting

Published 2026-07-26

Chain-of-thought prompting is instructing a model to work through its reasoning step by step before giving an answer. It measurably improves accuracy on arithmetic, logic and multi-step analysis, because generating intermediate steps gives the model more computation to reach a conclusion. It does not help short creative work, and often makes it worse.

What is chain-of-thought prompting?

Chain-of-thought prompting is asking a model to show its working before committing to an answer, rather than producing the answer directly. In its simplest form it is a single added sentence: "Work through this step by step before answering."

The mechanism is mechanical rather than mystical. A model generates one token at a time, and each generated token is available as input for the next. Producing intermediate reasoning gives the model more steps in which to compute, and more of its own prior conclusions to build on. An answer produced immediately has had far less computation behind it.

When does chain-of-thought actually help?

Task type	Helps?	Why
Arithmetic and unit conversion	Strongly	Intermediate values can be carried and checked rather than guessed.
Logic and constraint puzzles	Strongly	Each constraint can be applied in turn instead of all at once.
Multi-step analysis	Yes	Forces the evidence to be stated before the conclusion rests on it.
Debugging code	Yes	Tracing execution surfaces the fault rather than pattern-matching a fix.
Classification	Sometimes	Helps on ambiguous boundaries, wasteful on clear-cut cases.
Short creative writing	No	Deliberation shows up as stilted, over-explained prose.
Simple extraction	No	Adds cost and latency to a task with no reasoning in it.

The pattern is consistent: chain-of-thought helps where there are genuine intermediate steps that could each be got wrong. It hurts where the task is a single judgement, because the model then narrates a decision it had already made.

How do you write a chain-of-thought prompt?

The weak version is the famous one: "Let's think step by step." It works, but it is vague about which steps.

The stronger version names them:

Before answering:
1. Restate the problem in your own words.
2. List what you know and what you are assuming.
3. Consider at least two approaches and say why you chose one.
4. Work through the chosen approach.
5. State your answer, then check it against step 2.

Show the reasoning that actually drives your conclusion, not a tidy
summary written after the fact.

That last line matters more than it looks. Without it, models often produce reasoning that is a post-hoc justification of an answer already decided — which looks like chain-of-thought and provides none of the benefit.

Do reasoning models still need chain-of-thought prompts?

Largely no, and this is the biggest recent change in the advice. Reasoning models are trained to deliberate internally before answering, so instructing them to think step by step is redundant and can interfere — you end up with visible deliberation on top of the internal kind, which is slower without being better.

What still helps with reasoning models is everything that is not about the reasoning process: a clear statement of the goal, the constraints that bound an acceptable answer, and the criteria the result will be judged against. Give them a well-specified problem and let them choose how to work through it.

A practical rule: if the model shows you a thinking or reasoning phase before it answers, delete your "think step by step" instruction and spend the space on constraints instead.

How do you get the reasoning without seeing it?

Often you want the accuracy benefit but not a page of working in the output. Ask for both, separated:

Work through the problem inside <thinking> tags. Then give your final
answer inside <answer> tags, written for someone who has not seen the
thinking. The answer must stand alone.

Your application then extracts the <answer> block and discards the rest. You pay for the reasoning tokens either way — that is what buys the accuracy — but the user never sees them.

What is self-consistency and when is it worth the cost?

Self-consistency runs the same prompt several times independently and takes the most common answer. It works because errors in reasoning tend to be varied while correct reasoning tends to converge — five runs that agree are considerably more trustworthy than one.

The cost is linear in the number of samples, so it suits verification steps rather than every request. It also needs answers discrete enough to compare: a number, a category, a yes or no. Five essays cannot be majority-voted.

Common questions

Does "let's think step by step" actually work?: It does measurably improve accuracy on multi-step problems with older non-reasoning models, which is why the phrase became well known. Naming the specific steps you want works better, and with reasoning models the instruction is redundant.
Does chain-of-thought prompting cost more?: Yes. The reasoning is generated output, so you pay for those tokens and wait for them. That is precisely the trade: more computation spent on the problem, in exchange for money and latency. It is worth it on hard problems and wasteful on easy ones.
Can chain-of-thought make the answer worse?: On short creative or stylistic tasks, yes. The deliberation shows up in the prose as over-explanation and hedging. It can also let a model talk itself out of a correct first instinct on simple factual questions.
Is the reasoning a model shows its real reasoning?: Not necessarily. Generated reasoning is text produced by the same process as any other output, and it can be a plausible narrative rather than a faithful account. It is still useful — both because generating it improves accuracy and because it exposes checkable intermediate claims — but it should be read as a hypothesis, not a transcript.
Should I use chain-of-thought with reasoning models?: Usually not. They already deliberate internally, so the instruction is redundant and can interfere. Spend the space on a precise goal and clear constraints instead.