Prompt injection
Prompt injection is an attack in which instructions hidden inside content a model processes cause it to ignore its original instructions.
It works because a model sees one undifferentiated stream of text: it cannot inherently tell your instructions from a sentence inside a web page it was asked to summarise. Delimiting untrusted content and stating that instructions inside it must be treated as data reduces the risk but does not eliminate it.
Related terms
- Indirect prompt injection
- Indirect prompt injection is prompt injection delivered through content the model retrieves itself — a web page, an email, a document — rather than typed by the user.
- Jailbreak
- A jailbreak is a prompt crafted to make a model bypass its own safety training and produce output it would normally refuse.
- Delimiter
- A delimiter is a marker such as an XML tag, a row of dashes, or triple backticks that separates one part of a prompt from another.
More on safety
See the full glossary, read the guides, or put it into practice in the prompt builder.