AI Skill Report Card
Analyzing Wayback Machine CDX Data
Quick Start14 / 15
Given a table of Wayback Machine captures with columns like URL | MIME Type | From | To | Captures | Duplicates | Uniques, immediately produce:
- A summary count (total rows shown vs. total filtered/unfiltered entries, per footer text)
- Breakdown by MIME type
- Date range of the dataset
- Any notable outliers (highest capture counts, unknown MIME types, single-day-only entries)
Recommendation▾
Add an example showing a 'bad' or incorrect analysis (e.g., misreading the footer) to illustrate the pitfall contrast more concretely
Workflow13 / 15
Progress:
- Parse footer/header for total entry counts ("Showing X to Y of Z entries (filtered from N total)")
- Identify the URL pattern (base endpoint + query parameter variations, e.g.
PATTERNID=) - Tally MIME types (flag
unk/unknown separately) - Compute date span (earliest
From, latestTo) across the dataset - Sum Captures, Duplicates, Uniques columns
- Flag rows where
Captures > 1(these show re-crawl/change activity) vs. single-capture rows - Note any distinct identifier values (e.g., PATTERNID hashes) if the question concerns specific resources
- Present findings concisely; don't dump the full table back unless asked
Recommendation▾
Include a brief note on handling malformed or missing footer text when totals can't be determined
Examples17 / 20
Example 1: Input: A CDX table like the one above, question: "How many unique MIME types are there?" Output:
2 MIME types:
- application/weaving-information-format (48 rows)
- unk (2 rows)
Example 2: Input: Same table, question: "Which URLs were captured more than once?" Output:
6 URLs had multiple captures (Captures > 1), all with 1 duplicate + re-crawl later in 2005:
- PATTERNID=2B071234AB1AB476 (Jul 5 2004 → Aug 19 2004)
- PATTERNID=5D6735991EE56CE5 (Apr 28 2005 → Sep 1 2005)
- PATTERNID=0022F14284684FFD (Apr 16 2005 → Sep 12 2005)
- PATTERNID=39D74361F38A1818 (Jun 28 2004 → Aug 15 2004)
- PATTERNID=63CE8EBB9999F919 (Apr 7 2005 → Sep 6 2005)
- PATTERNID=76DDC3128C59E420 (Apr 27 2005 → Aug 31 2005)
All others are single-capture (Captures=1, Duplicates=0, Uniques=1).
Example 3: Input: "How does this subset relate to the full dataset?" Output:
Footer states: "Showing 1 to 50 of 218 entries (filtered from 10,000 total entries)."
This means: 10,000 total CDX records exist for the site/query; a filter
(likely restricted to the DownloadWIF.aspx path) narrowed it to 218 matching
entries; only the first 50 are rendered in this table.
Recommendation▾
Consider a small template for the final summary output format to standardize presentation across varied queries
Best Practices
- Always surface the filtering context from table footers — it changes the meaning of any count you report.
- Treat
unkMIME type as a distinct, flaggable category, not noise to ignore. - When a URL has no query string (e.g., the bare
DownloadWIF.aspx), treat it as an aggregate/index row, not a specific resource — call this out separately from per-ID rows. - Use
Captures - Uniques = Duplicatesas a sanity check when numbers are given. - When summarizing many near-identical rows (same MIME type, Captures=1), compress them into a count rather than listing each one.
- Preserve exact hex IDs (e.g., PATTERNID values) verbatim — don't truncate or re-case them.
Common Pitfalls
- Don't assume "Showing 1 to 50 of 218" means 218 is the grand total of the archive — check for "filtered from N total" which indicates a larger unfiltered population.
- Don't conflate
Captures(total crawl attempts) withUniques(distinct content versions) — a row can have multiple captures but only one unique version. - Don't ignore the base/parameterless URL row when counting "distinct resources" — decide explicitly whether to include or exclude it based on the question.
- Don't recompute totals by summing only the visible rows when the table is explicitly truncated (e.g., "1 to 50 of 218") — flag that the sum is partial.