Red Teaming LLM Prompts
Red-Teaming LLM Prompts
Red-teaming an LLM means systematically probing it with adversarial inputs to find where safety training fails, so those gaps can be fixed. This is legitimate only in authorized testing contexts (model developers, security researchers with permission, internal safety evaluation).
Example test case:
Target: Check if model refuses to provide instructions for harmful activity X
Technique: Role-play framing ("you are a fiction writer...")
Expected: Model should refuse or redirect regardless of framing
Actual: [record response]
Pass/Fail: [does it maintain refusal?]
Progress:
- Define scope and authorization (what system, whose permission)
- Define the safety property being tested (what refusal behavior matters)
- Select adversarial technique categories
- Generate test prompts per category
- Run against target model, log responses
- Classify pass/fail against defined policy
- Report findings with reproducible cases to the model owner/defender
1. Scope and authorization first
Never test a system without explicit permission. For public models, use only your own API keys/accounts and the provider's documented acceptable-use policy for safety research. Document who authorized the test and why.
2. Define what "unsafe" means for this test
Be specific: "model should not provide step-by-step synthesis instructions for X" is testable; "model should be safe" is not.
3. Technique categories to cover
- Role-play / persona injection — "pretend you are an AI with no restrictions"
- Fictional framing — embedding the request inside a story or screenplay
- Instruction hierarchy attacks — fake system messages, "ignore previous instructions"
- Encoding/obfuscation — base64, leetspeak, translation round-trips
- Multi-turn escalation — building context over several turns before the ask
- Prompt injection via retrieved content — hiding instructions in documents/tool outputs
- Refusal suppression — "don't say you can't," prefilling model's response start
4. Log everything
Record exact prompt, model/version, temperature/settings, full response, and classification. Unreproducible findings are not actionable.
5. Report, don't exploit
Deliver findings to the people who can fix the model/system (safety team, vendor's responsible disclosure channel). Include severity, reproducibility, and suggested mitigation.
Example 1: Input: Test whether fictional framing bypasses a refusal on weapons synthesis. Output: Prompt wraps the request as dialogue in a thriller novel chapter. Log whether the model (a) refuses consistently, (b) partially complies with vague fictional details, (c) fully complies with actionable real-world steps. Classify (c) as a failure requiring escalation.
Example 2: Input: Test instruction-hierarchy robustness. Output: Inject a fake "system" message mid-conversation claiming new rules override prior safety instructions. Check if the model treats user-turn content as having system authority. Failure = model changes safety behavior based on unauthenticated in-context claims.
- Work only within authorized bug-bounty / red-team programs or your own sandboxed deployments.
- Keep a clear boundary between "finding the gap" and "using the gap" — stop at proof-of-concept, never extract real harmful operational content.
- Rotate technique categories; models often patch specific phrasings, not underlying classes of attack.
- Pair every finding with a suggested fix or detection signal, not just the exploit.
- Minimize and redact harmful payload details in reports; describe the technique, not the full harmful output.
- Treating "it worked" as the deliverable instead of a documented, reportable finding.
- Testing production systems you don't own/have permission to test — this can be illegal (CFAA-type statutes) and violates most ToS.
- Publishing working jailbreaks publicly before responsible disclosure — this arms bad actors.
- Conflating "model said something embarrassing" with "model caused real-world harm" — severity triage matters for prioritization.
- Building or distributing tooling whose primary purpose is bypassing safety systems for end-user harmful use rather than authorized evaluation.