AI Skill Report Card

Auditing Vendor Artifact Pipelines

A-86·Aug 26, 2026·Source: Extension-selection
13 / 15

When someone claims to have "reviewed the vendor" or "understood the architecture," don't accept the claim — check the read order and read completeness against source hierarchy:

1. Does normative doc (SYSTEM_INSTRUCTIONS.md-equivalent) exist? → read first, fully.
2. Does architecture-of-record doc exist (e.g. native-harness-spec.md)? → read second.
3. Were runtime/compiler source files read completely, or inferred from names?
4. Is there a discrepancy between summary docs (README) and normative docs?
   → normative always wins.
5. Was historical/proposed material (UPB, RFCs, future proposals) read
   BEFORE the current grounded implementation? → that's backwards, flag it.

If the answer to (3) is "inferred," stop and re-read the actual files before drawing architectural conclusions.

Recommendation
Add a concrete before/after example showing corrected code (e.g., replacing hardcoded path with resolver call) to make Example 3 fully concrete rather than descriptive
14 / 15

Progress:

  • Step 1: Identify the source hierarchy (normative spec > architecture-of-record > README/framing > current contracts > implementation files > schemas/tests > historical/proposed)
  • Step 2: For each file, record read status explicitly: complete / partial / not read / inferred
  • Step 3: Trace the artifact pipeline as the vendor actually defines it — list every named intermediate representation in order (e.g. source → IntentGraph → IR → SemanticPackage → CompilationContext → PromptPacket → ExecutionRecord)
  • Step 4: Identify every identity/authority mechanism enforced in code (hash checks, authority tier tables, conflict resolution order) — not just described in prose
  • Step 5: For each boundary object in the pipeline, ask: does the outer/integrating layer preserve and forward this object, or does it replace it with hand-rolled state?
  • Step 6: Flag any place the outer layer duplicates vendor-owned resolution logic (hardcoded paths, reimplemented lookups, independent reordering of authority)
  • Step 7: Reconcile discrepancies between summary docs and normative docs by deferring to normative source
  • Step 8: Restate the integration task as "expose/preserve/transport/re-consume native artifacts" rather than "build a new protocol"
Recommendation
Include a brief example of a 'good outcome' where prior work passed the audit, to contrast with the mostly-flag-focused examples
17 / 20

Example 1: Input: "I reviewed the vendor SDK by reading the README and skimming filenames in lib/, then designed our integration around 'call compiler, get output, send to model.'" Output: Flag as inference-based review, not read-based review. Require: locate the normative/adoption doc (not README) first. Re-derive the pipeline from actual source files (compiler, runtime, execution files), not from the assumed three-step shape. Check whether the compiler enforces a typed intermediate format (e.g. rejects raw input, requires a specific structured package format) — this alone falsifies the "call compiler, get output" mental model and reveals missing intermediate stages.

Example 2: Input: Two docs disagree: a README table says subsystem X does "artifact packaging," while the normative spec's canonical ID table says subsystem X does "dependency resolution." Output: Defer to the normative spec as authoritative. Explicitly note the discrepancy rather than silently picking whichever one was read first. Warn that any earlier work built on the README's characterization needs re-checking against the canonical vocabulary.

Example 3: Input: An outer CLI has code like path.join(vendorRoot, "references", id) to locate vendor resources. Output: Treat as architecturally suspect if the vendor exposes its own resolver (e.g. reference.resolve(logical_id)) that performs multi-layer lookup, containment checks, hashing, and ambiguity detection. Recommend replacing the hardcoded path with a call into the native resolver and consuming its returned binding/evidence, rather than maintaining a parallel fallback list.

Recommendation
Consider trimming some overlapping bullet points between Best Practices and Common Pitfalls to tighten the skill further
  • Read order mirrors authority order. Normative/adoption source first, architecture-of-record second, framing docs third, current contracts fourth, implementation fifth, schemas/tests sixth, historical/proposed material last. Proposed or future-facing material should never outrank the grounded current implementation in an investigation.
  • Record completeness per file. "Complete," "partial," or "not read" — never silently blend snippets, filename inventories, and prior knowledge into something that reads as "I've reviewed this."
  • Preserve source attribution. Don't merge multiple sources into one unattributed synthesis; keep them separately traceable, the same way the artifacts themselves carry source hashes/line spans.
  • Distinguish "lowering/interpretation" from "execution." A system may deliberately preserve certain constructs as "not lowered" / "not executed" rather than pretending to have handled them — this conservatism is a feature, not a gap.
  • Treat intermediate representations as semantically load-bearing, not debug scaffolding — if a compiler refuses raw input and demands a specific typed intermediate, that constraint is architecture, and any wrapper that skips the intermediate is violating the contract.
  • Identity/hash checks are the real authority mechanism. Prose describing a pipeline is a claim; a source_sha256 check or semantic_hash recomputation in code is the enforcement. Look for the code, not just the description.
  • Authority/conflict-resolution tables are normative inputs, not suggestions. A harness or wrapper should deliver the native compiled/ordered result faithfully rather than independently reordering or reinterpreting authority tiers.
  • When integrating a new outer layer, ask of every native boundary object: is it preserved and forwarded, or replaced with hand-written prose/JS/ad hoc state? That question is the actual integration audit.
  • Treating "I looked at the filenames and skimmed a README" as equivalent to having read the normative source.
  • Letting historical, proposed, or aspirational documents (RFCs, future-version proposals) shape the mental model of the current system before the current implementation has been read.
  • Picking whichever summary doc happens to be open, rather than checking it against the canonical/normative vocabulary when the two disagree.
  • Assuming a simple three-stage pipeline (input → compiler → output) when the real system enforces a multi-stage typed pipeline with explicit intermediate artifacts and refuses to shortcut it.
  • Reimplementing vendor-owned resolution/lookup logic (paths, aliases, fallback layers) in the outer harness instead of calling the vendor's own resolver and consuming its evidence.
  • Flattening a rich intermediate artifact (one containing hashes, authority snapshots, conflict records, verification rules, provenance) down to "just the text sent to the model," losing the audit trail.
  • Failing to explicitly flag discrepancies between docs, silently resolving them by preference rather than by authority hierarchy.
0
Grade A-AI Skill Framework
Scorecard
Criteria Breakdown
Quick Start
13/15
Workflow
14/15
Examples
17/20
Completeness
18/20
Format
14/15
Conciseness
13/15