AI Skill Report Card
Generated Knowledge Prompting
Markdown--- name: generated-knowledge-prompting description: Designs two-stage prompts that first elicit relevant background knowledge from an LLM, then use that knowledge to produce a grounded final answer. Use when a question requires specialized/factual knowledge, when hallucination risk is high, when the LLM's answer quality seems shallow, or when building prompt pipelines that need explainable intermediate reasoning before the final response. ---
Quick Start14 / 15
Pythonimport openai def generate_response(prompt, max_tokens=500, temperature=0.7): response = openai.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}], max_tokens=max_tokens, temperature=temperature, ) return response.choices[0].message.content.strip() question = "Explain the concept of quantum entanglement and its potential applications." # Stage 1: elicit knowledge knowledge_prompt = f"""Generate 4-6 concise, factual bullet points of background knowledge relevant to answering this question. Do not answer the question yet. Question: {question}""" knowledge = generate_response(knowledge_prompt, temperature=0.3) # Stage 2: answer using the generated knowledge answer_prompt = f"""Using ONLY the knowledge below, answer the question. If the knowledge is insufficient, say so explicitly. Knowledge: {knowledge} Question: {question} Answer:""" answer = generate_response(answer_prompt, temperature=0.5)
Recommendation▾
Example 2 and 3 are more abstract/descriptive than Example 1 (which has concrete JSON); add actual generated text output for all examples to maintain consistency
Workflow14 / 15
Progress:
- Step 1: Define the task and the exact question/prompt to answer
- Step 2: Decide if knowledge generation is warranted (see Best Practices)
- Step 3: Craft the knowledge-elicitation prompt (specify format, scope, keywords)
- Step 4: Generate knowledge at low temperature (0.2–0.4) for determinism
- Step 5: Inspect generated knowledge for relevance/accuracy before reuse
- Step 6: Construct final prompt combining question + generated knowledge
- Step 7: Generate final answer, instructing the model to rely on the knowledge
- Step 8: Validate final answer against the knowledge for consistency (catch hallucinations)
- Step 9: Iterate on prompt wording if knowledge is irrelevant, repetitive, or wrong
Recommendation▾
Consider showing a 'bad outcome' example where skipping validation leads to a hallucinated knowledge snippet propagating into the final answer
Examples15 / 20
Example 1: Structured JSON knowledge Input:
Question: What are the benefits of using renewable energy sources?
Knowledge prompt: "Generate relevant knowledge as a JSON array of objects
with keys 'source' and 'benefits'."
Output:
JSON[ {"source": "Solar", "benefits": "Low operating cost, scalable, reduces emissions"}, {"source": "Wind", "benefits": "High energy yield, minimal land footprint when offshore"} ]
Final answer prompt feeds this JSON in, parsed via json.loads (with try/except
for malformed output), and the model composes a final narrative answer.
Example 2: Historical reasoning Input: "What were the main causes of World War I?" Output: Stage 1 generates knowledge on alliances, militarism, nationalism, the assassination of Archduke Franz Ferdinand; Stage 2 synthesizes these into a structured causal answer (political, military, social factors).
Example 3: Insufficient knowledge detection Input: "What is the current stock price of XYZ Corp?" Output: Generated knowledge stage should return something like "No reliable static knowledge available — this requires real-time data," prompting the final stage to decline rather than hallucinate a number.
Recommendation▾
The checklist workflow is strong, but could briefly note how to detect knowledge insufficiency programmatically (e.g., keyword matching for 'insufficient')
Best Practices
- Use it selectively. Reserve for tasks needing specialized/niche knowledge, multi-hop reasoning, or high accuracy. Skip for trivial factual lookups — it adds latency, tokens, and cost for no benefit.
- Separate temperature settings. Lower temperature (0.2–0.4) for the knowledge-generation stage (favor determinism/factuality); moderate temperature (0.5–0.7) for the final answer if some fluency/creativity is desired.
- Constrain format. Ask for bullet points, numbered facts, or JSON schemas to make the generated knowledge easy to parse, audit, and reuse downstream.
- Instruct explicit groundedness. In the final prompt, say "use only the knowledge above" and "state if insufficient" to reduce hallucination.
- Provide few-shot examples of good knowledge snippets when the domain is specialized, to steer format and relevance.
- Combine with other techniques. Layer with Chain-of-Thought (explain reasoning over the generated knowledge) or few-shot prompting (show sample knowledge+answer pairs) for compounding gains.
- Budget tokens and cost. Two LLM calls per question double latency/cost — cache generated knowledge when the same question/topic recurs.
- Always validate. Spot-check generated knowledge against trusted sources, especially for high-stakes domains (medical, legal, financial).
Common Pitfalls
- Don't skip validation — generated knowledge can itself be hallucinated; blindly trusting it just moves the hallucination earlier in the pipeline.
- Don't use vague elicitation prompts ("tell me about X") — they produce generic, low-utility knowledge. Be specific about scope, format, and keywords.
- Don't ignore malformed structured output — LLMs frequently emit invalid JSON; always wrap parsing in error handling with a retry or fallback.
- Don't apply this to simple queries — the added latency/cost isn't justified when the model's direct answer is already reliable.
- Don't let final-stage prompt drop the "use only this knowledge" constraint — without it, the model may ignore the generated knowledge and hallucinate anyway.
- Don't treat this as a substitute for retrieval/grounding — generated knowledge still comes from the model's internal parameters, not a verified external source; for critical facts, pair with real retrieval (RAG).