Should I paste my whole document into the prompt?

Only if the task genuinely requires reasoning across all of it. Otherwise select the relevant sections. Stuffing costs more, responds slower, and reduces recall of anything sitting in the middle of the context.

Where should the instruction go with a long document?

Both before and after. Stating the task first means the model reads the document knowing what it is looking for; repeating it at the end means the instruction is the most recent thing before it answers. This is the one case where repetition earns its tokens.

Why does the model miss information in the middle of a long document?

Lost in the middle: recall is measurably better near the start and end of a long context than in the middle. Where the important material sits matters, which is why the most important document should never be placed mid-stack.

Is RAG better than a long context window?

They solve overlapping problems. Retrieval is cheaper, faster, and keeps attention on relevant material; long context is simpler and handles reasoning that spans a whole document. Retrieval wins for question-answering over a corpus; long context wins when the task genuinely needs everything at once.

How do I stop a long conversation losing the plot?

Put the rules that must not slip in the system prompt, restate the current goal periodically, and start fresh when the topic changes. Carrying twenty turns of irrelevant history is context stuffing in a different form.

Prompting with long documents and large context

Published 2026-07-26

Models recall information at the start and end of a long context more reliably than in the middle, an effect known as lost in the middle. Selecting the relevant few hundred words usually beats supplying tens of thousands, and where you must supply a lot, structure and ordering determine what actually gets used.

Does a bigger context window mean you should use it?

No, and this is the most consequential misunderstanding about long-context models. A window of hundreds of thousands of tokens is a capacity limit, not a target. Filling it costs more, responds slower, and reliably degrades recall of anything that is not near an edge.

Context stuffing — pasting everything and hoping the model finds the relevant part — is a common and expensive habit. Selecting the right 500 words beats supplying the wrong 50,000, on every axis including accuracy.

What is the lost-in-the-middle effect?

Lost in the middle is the observed tendency for models to recall information placed near the beginning or end of a long context more reliably than information in the middle. Retrieval accuracy across a long document is not flat — it sags.

The practical consequences are direct:

When supplying several documents, do not place the most important one in the middle of the stack.
A critical instruction buried mid-prompt is the one most likely to be ignored.
"Find the relevant passage in this 200-page document" is a task with a known failure mode; do the retrieval yourself where you can.

How should a long-document prompt be structured?

Put the task before the document, and repeat it after. This is the one case where stating the instruction twice is worth the tokens: the model knows what it is looking for while reading, and the instruction is the most recent thing before it answers.

<task>
Extract every commitment with a date attached, and note who made it.
</task>

<document>
[50 pages of meeting transcripts]
</document>

<rules>
- Only commitments with an explicit date
- Quote the sentence containing each commitment
- If nobody is named, record the speaker as null
</rules>

Now extract the dated commitments from the document above.

Delimiting the document also matters for a second reason: without a clear boundary, a sentence inside it can read as an instruction — see prompt injection.

When should you split rather than stuff?

When the task can be applied to each part independently, splitting is almost always better. Processing ten sections separately and combining the results gives you ten focused tasks instead of one diluted one — and each result can be checked.

Situation	Approach
Extract every X from a large document	Split into sections, extract from each, merge
Summarise a large document	Summarise sections, then summarise the summaries
Answer one question from a large corpus	Retrieve the relevant passages first (RAG)
Compare two long documents	Extract structured facts from each, then compare the structures
Reason across the whole document at once	Genuinely needs long context — keep it in one call

Only the last row justifies filling the window. It is a real case, and it is rarer than it feels.

How do you keep a long conversation on track?

Long conversations hit the same limits as long documents, plus prompt drift: early instructions get outweighed by recent turns, so the persona slips and formatting rules lapse.

Put non-negotiable rules in the system prompt, and at the end of it.
Periodically re-state the current goal in a user turn — a running summary beats relying on twenty turns of history.
Start a fresh conversation when the topic genuinely changes. Carrying irrelevant history is context stuffing by another route.
For agents, keep a compact explicit state object rather than expecting the model to remember what happened.

Common questions

Should I paste my whole document into the prompt?: Only if the task genuinely requires reasoning across all of it. Otherwise select the relevant sections. Stuffing costs more, responds slower, and reduces recall of anything sitting in the middle of the context.
Where should the instruction go with a long document?: Both before and after. Stating the task first means the model reads the document knowing what it is looking for; repeating it at the end means the instruction is the most recent thing before it answers. This is the one case where repetition earns its tokens.
Why does the model miss information in the middle of a long document?: Lost in the middle: recall is measurably better near the start and end of a long context than in the middle. Where the important material sits matters, which is why the most important document should never be placed mid-stack.
Is RAG better than a long context window?: They solve overlapping problems. Retrieval is cheaper, faster, and keeps attention on relevant material; long context is simpler and handles reasoning that spans a whole document. Retrieval wins for question-answering over a corpus; long context wins when the task genuinely needs everything at once.
How do I stop a long conversation losing the plot?: Put the rules that must not slip in the system prompt, restate the current goal periodically, and start fresh when the topic changes. Carrying twenty turns of irrelevant history is context stuffing in a different form.