AI Skill Report Card

Extracting Literature Requirements

A-85·Nov 9, 2026·Source: Web
13 / 15

Given one PDF and a research question, produce exactly three outputs:

  1. Study profile (keyword-level)
  2. Ranked requirement-candidate table (≤15 rows)
  3. Lists: contradictions/limitations, unusable parts, uncertainties

No design principles. No evaluation beyond the text. No knowledge outside the PDF. If the PDF is unreadable or empty, state this immediately and stop.

Recommendation▾
Add a third example showing an 'unusable part' or 'uncertainty' flag scenario in full, not just requirement candidates, to demonstrate the complete output format.
15 / 15
Progress:
- [ ] Step 1: Confirm PDF is readable; if not, report and stop
- [ ] Step 2: Read full text; build study profile (type, sample, context, main finding)
- [ ] Step 3: Extract candidate statements tied to the research question's layers
- [ ] Step 4: For each candidate, locate verbatim sentence + section/page
- [ ] Step 5: Classify type (finding / recommendation / implemented feature / conjecture)
- [ ] Step 6: Assign layer (1-8), write mechanism/rationale from text only
- [ ] Step 7: Assess transferability (direct vs indirect) with brief justification
- [ ] Step 8: Rank candidates by relevance to RQ; cap at 15, note remainder by layer
- [ ] Step 9: Collect contradictions/limitations (verbatim) stated by authors
- [ ] Step 10: List unusable parts (half-sentence each) and flag own uncertainties

Step 2 detail: Study Profile

Extract only what's stated, in keyword form:

  • Type: empirical / conceptual / review / design study / experiment — "not in text" if absent
  • Sample: N, population, context of study — "not in text" if absent
  • Context: setting (e.g., university course, lab study) — "not in text" if absent
  • Main finding: two sentences, taken from or tightly paraphrased from the text, citing location

Step 3-6 detail: Candidate Extraction

For each candidate statement:

  • Phrase as "The system should ..., so that ..." — but only if this follows logically from the source sentence. Do not invent justifications not supported by text.
  • Pull the exact verbatim sentence (max one sentence) that supports it. If the supporting evidence is only in the abstract, mark "abstract only".
  • Classify Type:
    • finding: empirically observed result
    • authors' recommendation: explicit prescriptive statement by authors
    • implemented feature: a system/tool feature described as built and used
    • conjecture: author speculation, future work, hypothesis not tested
  • Assign exactly one Layer (1-8 per the scheme below). If ambiguous, pick primary layer and flag in uncertainty list.
  • Mechanism/rationale: state only the causal or explanatory logic the paper itself gives (e.g., "increases awareness because X"). If paper gives no mechanism, write "not in text".
  • Transferability:
    • direct: applies straightforwardly to AI-chat self-regulation dashboards
    • indirect: applies to a different context (e.g., general SRL, LMS dashboards); briefly state the gap

Layer reference:

#Layer
1What is mirrored (content of reflection)
2Data source (what's tracked/logged)
3Presentation/feedback (visualization, format)
4Prompting for reflection (nudges, questions)
5Adaptivity/dosing (personalization, frequency/timing)
6Autonomy/engagement (user control, motivation)
7Usage behavior/needs (how/why users act)
8Other/unexpected

Step 8: Ranking & Overflow

Rank by direct relevance to the RQ (self-regulated AI use, metacognitive processes, dashboard design). If more than 15 strong candidates exist, output the top 15 and append: "N additional candidates exist, primarily in layers [X, Y, Z]."

Step 9-10: Contradictions, Limitations, Unusable Parts, Uncertainties

  • Contradictions/limitations: verbatim sentence + brief note on what it contradicts or limits (e.g., re: dashboards generally, re: self-regulation claims, re: AI-chat context specifically)
  • Unusable parts: half-sentence summary of sections not relevant to RQ (e.g., "Section 4.2: unrelated statistical validation of survey instrument")
  • Uncertainties: flag your own classification judgment calls (layer assignment ambiguity, borderline type classification, paraphrase vs. verbatim boundary cases)
Recommendation▾
Consider trimming the layer reference table explanation slightly or moving detailed step instructions into a more compact reference format to tighten overall length.
17 / 20

Example 1: Input: PDF states: "Students who received weekly summaries of their query patterns reported increased awareness of over-reliance on AI for factual recall (p<.05)." Output:

No.RequirementOriginal sentenceLocationTypeLayerMechanismTransferability
1The system should provide periodic summaries of query patterns, so that students become aware of over-reliance tendencies."Students who received weekly summaries of their query patterns reported increased awareness of over-reliance on AI for factual recall (p<.05)."Sec. 4.1, p.12finding3Awareness increases via periodic aggregated feedback (per text)direct

Example 2: Input: PDF only discusses general SRL theory in education (not AI-specific), stating: "Monitoring is a prerequisite for effective control in self-regulated learning cycles." Output:

No.RequirementOriginal sentenceLocationTypeLayerMechanismTransferability
2The system should support monitoring before enabling control actions, so that users follow an effective self-regulation sequence."Monitoring is a prerequisite for effective control in self-regulated learning cycles."Sec. 2, p.3authors' recommendation1Sequential dependency in SRL cycle (per text)indirect: general SRL theory, not AI-chat specific — gap: no AI chat context tested
Recommendation▾
Clarify how ties in ranking are resolved or what counts as 'strong' candidate when capping at 15, to reduce ambiguity in edge cases.
  • Always quote exactly; never paraphrase into the "Original sentence" column.
  • If a mechanism is implied but not stated, write "not in text" rather than inferring.
  • Keep "so that" clauses tethered to text-stated outcomes — don't add plausible-sounding benefits the paper doesn't claim.
  • Treat abstract-only evidence as lower confidence; mark clearly.
  • When the paper discusses AI-supporting-learning (not self-regulated use of AI) treat as indirect transferability or unusable, per the stated definitions.
  • Err toward "not in text" over filling gaps with domain knowledge — this is the core constraint of the task.
  • Do NOT bring in Zimmerman's SRL model or Gregor's DP anatomy to fill gaps in what the paper says — those are for later synthesis steps, not this extraction.
  • Do NOT formulate design principles — only requirement candidates.
  • Do NOT evaluate quality, validity, or generalizability of the study beyond what authors themselves state as limitations.
  • Do NOT merge multiple sentences into "Original sentence" — one sentence max, verbatim.
  • Do NOT assign multiple layers to one candidate — pick the primary layer, note ambiguity separately if needed.
  • Do NOT silently drop candidates that contradict the thesis direction — these belong in the table (marked appropriately) and/or contradictions list.
0
Grade A-AI Skill Framework
Scorecard
Criteria Breakdown
Quick Start
13/15
Workflow
15/15
Examples
17/20
Completeness
18/20
Format
14/15
Conciseness
12/15