Prompt evaluation
Prompt evaluation is testing a prompt against a fixed set of inputs and grading criteria to find out whether a change actually improved it.
Without evaluation, prompt work is guesswork dressed as craft — a change that looks better on one example is assumed to be better everywhere. Even ten held-out test inputs and a written rubric turn that into something measurable.
Related terms
- LLM as judge
- LLM as judge is using one language model to grade another model’s output against a written rubric.
- Golden dataset
- A golden dataset is a fixed set of inputs with known-good outputs, used as the benchmark for judging whether a prompt change is an improvement.
- Overfitting a prompt
- Overfitting a prompt is tuning it so tightly to a handful of test inputs that it performs worse on the general case.
More on evaluation
See the full glossary, read the guides, or put it into practice in the prompt builder.