Overfitting a prompt
Overfitting a prompt is tuning it so tightly to a handful of test inputs that it performs worse on the general case.
It happens when a prompt is iterated against the same three examples until they pass. Holding back inputs that were never used during tuning is the only reliable way to notice it.
Related terms
- Prompt evaluation
- Prompt evaluation is testing a prompt against a fixed set of inputs and grading criteria to find out whether a change actually improved it.
- Few-shot prompting
- Few-shot prompting is giving a model several worked examples of a task inside the prompt so it can infer the pattern and apply it to new input.
More on failure modes
See the full glossary, read the guides, or put it into practice in the prompt builder.