Refining Knowledge Intensive QA with Internal External Retrieval
Refining Knowledge-Intensive QA with Internal-External Knowledge Interaction
Core pipeline for a multi-choice QA query:
- Extract named entities from the question with an off-the-shelf NER model.
- For each entity, prompt the LLM:
"Tell me something about [entity]"→ collect generated internal knowledge sentences, concatenate intoIK. - Use
[query : IK : candidate_KB_sentence]as input to a cross-encoder retriever that scores each external KB triple (converted to natural language) for relevance; keep top-k asEK. - Feed
[query, IK, EK]into the LLM reader to produce the final answer via standard cross-entropy-trained generation (or scoring over answer candidates).
This closes the "query → needed knowledge" gap: raw queries often share only superficial lexical overlap with the KB facts that are actually required (e.g., query mentions "steel", needed fact is "metal is a thermal conductor"). Internal LLM knowledge bridges that gap before retrieval happens.
Progress checklist for building this system:
- Task formulation: Define the KB as a set of (entity, relation, entity) triples; define the QA task as selecting
argmax_{a in Candidates} P(a | q). - Entity extraction: Run NER over the query to identify key concepts/entities.
- Internal knowledge reflection: For each entity
e_i, generatek_i = LLM(Prompt + e_i). Concatenate sequentially intoIK. Do this for ALL extracted entities, not just the most salient one — coverage matters more than precision here. - KB pruning (optional but recommended at scale): Perform entity linking from query/IK entities to KB nodes; restrict search to a 2-hop subgraph to make retrieval tractable.
- Triple verbalization: Convert each (h, r, t) triple into a natural language sentence using relation-specific templates (e.g.,
(ice, HasProperty, cold)→ "Ice has the property of cold"). - External retrieval: Score each verbalized KB sentence with a cross-encoder over
[query : IK : candidate_sentence], using an MLP head on top for the similarity score. Select top-m sentences asEK. - Answer generation: Concatenate
query + IK + EKas reader input; train/fine-tune with cross-entropy loss over the answer tokens (or do candidate scoring for multi-choice). - Evaluation: Benchmark accuracy against both GNN-augmented LMs (e.g., QA-GNN, GreaseLM) and pure LLM/prompting baselines (e.g., UnifiedQA, Flan-T5, Generated Knowledge Prompting) on held-out dev/test splits.
- Ablations: Test variants — reader without IK, reader without EK, retrieval without IK conditioning — to isolate each module's contribution.
Example 1: Input: "Which of these would let the most heat travel through? A) a new pair of jeans. B) a steel spoon in a cafeteria. C) a cotton candy at a store. D) a calvin klein cotton hat" Process: Entities extracted → "jeans", "steel spoon", "cotton candy", "cotton hat". Internal reflection on "steel" yields something like "Steel is a metal alloy, often used in cutlery, conducts heat well." This bridges to the KB fact "Metal is a thermal conductor" which pure lexical retrieval on the raw query would likely miss (query says "steel", not "metal" or "conductor"). Output: Retriever now surfaces "Metal is a thermal conductor" as top external knowledge; reader combines this with IK to select B.
Example 2:
Input: Build retrieval scoring function for candidate KB sentence K_i given query q and internal knowledge IK.
Output: s_i = MLP(CrossEncoder([q : IK : K_i])); rank all candidates by s_i; concatenate top-m as EK = Concat[K_(1) || K_(2) || ... || K_(m)].
- Prompt simplicity wins: A minimal template like "Tell me something about X" is sufficient to elicit useful parametric knowledge — no need for elaborate prompt engineering.
- Condition retrieval on IK, not just the raw query: Always concatenate internal knowledge with the query before scoring external candidates; this is the key mechanism that improves retrieval precision.
- Verbalize structured KB triples into natural sentences before feeding them to a text-based cross-encoder — raw triples don't match the embedding space of free text well.
- Prune the KB search space (e.g., entity-linking + k-hop subgraph) before scoring every triple; scoring the full KB is wasteful and noisy.
- Treat internal and external knowledge as complementary, not redundant: internal knowledge guides what to retrieve; external knowledge corrects/supplements what internal knowledge gets wrong or omits (hallucination mitigation runs both directions).
- Report on both dev and held-out/in-house test splits when official test labels are hidden, and be explicit about which data split convention you follow (cite prior work's split, e.g., Lin et al. 2019 for CommonsenseQA) so comparisons stay valid.
- Compare against both LM+GNN hybrid methods and pure LLM/prompting methods — knowledge-intensive QA has two competing paradigms, and a credible evaluation addresses both.
- Report model size (parameters) alongside accuracy when comparing to baselines — a smaller model matching or beating larger ones is a stronger result worth highlighting explicitly.
- Don't retrieve external knowledge using only the raw query — this reproduces the lexical gap problem the framework is designed to solve.
- Don't skip entity-level iteration — generating one blob of internal knowledge for the whole query (instead of per-entity) loses coverage on multi-entity questions.
- Don't feed raw KB triples directly into a text cross-encoder without verbalizing them into natural language templates first.
- Don't treat internal knowledge as ground truth — it can hallucinate; the external retrieval step exists specifically to correct it, so always fuse both rather than relying on internal knowledge alone for the final answer.
- Don't retrieve from the full, unpruned KB when a linked-entity subgraph restriction is available — this wastes compute and increases noise/distractor facts in top-k results.
- Don't omit ablations isolating IK-for-retrieval vs. IK-for-reading vs. EK-for-reading — reviewers and practitioners need to know which component actually drives gains.