Engineering Claude Prompts

Before touching prompt wording, confirm three things exist:

1. Success criteria — concrete, measurable definition of "good output" (e.g., "extracts all line items with correct totals," "refuses unsafe requests without over-refusing safe ones")
2. Eval method — a way to test outputs against those criteria empirically (test set + scoring, even if manual at first)
3. First draft prompt — something to iterate on

If any are missing, produce them first. Don't prompt-engineer against vibes.

Task: Summarize customer support tickets into 3 bullet points.

Success criteria:
- Captures the core issue (not just symptoms)
- Includes any explicit customer ask (refund, escalation, etc.)
- Under 50 words total
- No hallucinated details not in the ticket

Eval: 20 sample tickets, manual pass/fail scoring against criteria above.

Draft prompt v1:
"Summarize this support ticket in 3 bullet points: {ticket}"

Progress:

• Confirm success criteria and eval exist (build them if not)
• Diagnose why the current prompt underperforms
• Check if the failure is actually prompt-fixable (vs. model/latency/cost issue)
• Apply targeted technique(s) to address the specific failure
• Re-run eval, compare against baseline
• Iterate or stop when criteria met

Step 1: Diagnose before prescribing. Read actual failing outputs, don't guess. Categorize failures: ambiguous instructions? missing context? wrong format? reasoning errors? inconsistent behavior across similar inputs?

Step 2: Decide if prompt engineering is the right lever. Not every problem is a prompt problem:

• Slow responses → consider a different model, not a longer prompt
• High cost → consider a smaller/cheaper model or shorter context
• Task fundamentally exceeds model capability → consider decomposition (prompt chaining) or a stronger model, not more prompt tweaking
• Only proceed with prompt engineering when the failure is about instruction clarity, structure, or missing guidance

Step 3: Match technique to failure mode.

• Vague/inconsistent outputs → add clarity and explicit instructions (say exactly what you want, avoid ambiguity)
• Model misses format/edge cases → add examples (few-shot) covering those cases
• Complex multi-part output → use XML tags to structure input and requested output
• Wrong tone/perspective → use role prompting (system prompt defining persona/expertise)
• Errors in multi-step reasoning → prompt for step-by-step thinking before the final answer
• Task too large/multi-stage for one pass → break into a prompt chain (separate prompts, output of one feeds the next)

Step 4: Change one variable at a time. Modify the prompt, re-run the same eval set, compare pass rate directly to baseline. Don't bundle multiple technique changes in one iteration unless testing a combined hypothesis.

Step 5: Stop when criteria are met — don't over-engineer. Once eval passes consistently, stop adding instructions. Extra constraints often degrade robustness on inputs outside the test set.

Example 1: Input: Prompt produces inconsistent bullet counts (sometimes 2, sometimes 5) despite asking for "a few bullet points." Output: Diagnosed as ambiguity failure → replace "a few" with explicit "exactly 3 bullet points" and add an example showing the exact format. Re-eval confirms consistent 3-bullet output.

Example 2: Input: Classification prompt correctly labels obvious cases but fails on ambiguous edge cases (e.g., tickets that are both a complaint and a refund request). Output: Diagnosed as missing guidance for edge cases → add 2-3 few-shot examples specifically covering ambiguous/multi-label cases, plus explicit tie-breaking rule ("if multiple categories apply, choose the one mentioned first"). Re-eval shows edge-case accuracy improvement.

Example 3: Input: Prompt asks Claude to solve multi-step math word problems; final answers are frequently wrong. Output: Diagnosed as reasoning failure, not clarity failure → add instruction to think step-by-step in <thinking> tags before giving the final answer in <answer> tags. Re-eval shows improved accuracy since intermediate reasoning is now surfaced and correctable.

Example 4: Input: Team reports the assistant is "too slow" and wants a more detailed prompt to fix it. Output: Diagnosed as NOT a prompt engineering problem — latency is a model/infrastructure concern. Recommend evaluating a faster model variant instead of expanding the prompt.