AI Skill Report Card

Auditing Third Party Cyber Risk Evidence

A90·Nov 13, 2026·Source: Web

Auditing Third-Party Cyber Risk Evidence

14 / 15

Given supplier name, scope, and raw evidence artifacts, produce:

  1. Evidence & Findings Traceability Matrix (Markdown table)
  2. Granular Metrics Breakdown (counts, percentages, deltas, EOL risks, unverified coverage)
  3. Supplier Verification Email (collaborative, evidence-grounded, in the input's language)

Never summarize vendor-provided summaries. Parse raw rows/log lines directly. If any archive is unreadable, mark it "Unverified Coverage due to Unreadable Submission" and count 0% for that segment.

Recommendation▾
Fix the truncated final sentence under Best Practices ('When vendor summary and raw data disagree') to complete the thought
DomainSLA ThresholdKey Sub-Check
Vulnerability Management≥95% fleet complianceUnresolved Critical (9.0–10.0) / High (7.0–8.9) ≤5% of in-scope fleet
Endpoint Protection (AV/EDR)≥95% fleet complianceAgent health, signature age <7 days, stale/offline tracking
Patch Management≥95% fleet complianceOS cumulative updates AND third-party runtimes/apps
IAM (physical + logical)≥90% complianceDeactivation ≤24h from HR termination effective time, all domains/directories
Network & Perimeter0 toleranceNo unmanaged/unauthorized/consumer-grade hardware in production or Secure Workspace

If custom thresholds are supplied in the input, use those instead and state explicitly which baseline set is in effect.

15 / 15

Progress:

  • Step 1: Intake scope — confirm Supplier Name, Scope/Site/Geography, and inventory every submitted artifact (filename, format, sheet names)
  • Step 2: Triage artifacts — flag unreadable/corrupted/unsupported files immediately; do not attempt to infer their contents
  • Step 3: Parse vulnerability management evidence — compute fleet %, isolate Critical/High unresolved findings, identify root cause (scan scope gaps, agent failures)
  • Step 4: Parse endpoint protection evidence — compute agent health %, flag signatures >7 days old, flag devices reported online with no scan/update in 30 days as "stale"
  • Step 5: Parse patch management evidence — separate OS cumulative patch compliance from third-party app/runtime/OEM driver patch compliance; identify pipeline root cause (e.g., SCCM covering OS only)
  • Step 6: Parse IAM evidence — for every termination record, compute [Deactivation Timestamp] − [Effective HR Termination Timestamp] in hours/days; check all directories/forests for simultaneity; flag missing log trails as visibility risks, not compliance
  • Step 7: Parse network/perimeter evidence — identify any unmanaged, unauthorized, or consumer-grade devices in scope
  • Step 8: Identify infrastructure lifecycle risks — flag EOL/near-EOL OS on critical infrastructure (DCs, DNS, etc.) independent of scanner flags
  • Step 9: Build the Evidence & Findings Traceability Matrix — one row per distinct finding, cite exact source file/sheet
  • Step 10: Build the Granular Metrics Breakdown
  • Step 11: Draft the Supplier Verification Email — same language as input data, collaborative tone, 5-business-day ask
  • Step 12: State all assumptions and data gaps explicitly; never infer compliance from absence of data
Recommendation▾
Add 1-2 more concrete examples covering network/perimeter and EOL infrastructure scenarios to balance the IAM/patch-heavy examples
  • Deep artifact parsing only. Ignore vendor ReadMe/summary claims. Cross-reference raw rows against stated percentages — if a vendor summary says 97% but raw data supports 91%, report the computed 91% and note the discrepancy as a finding itself.
  • Unreadable archives → 0% auditable for that coverage segment. Label exactly: "Unverified Coverage due to Unreadable Submission".
  • Root-cause, not just symptom. For patch gaps, explicitly state whether the pipeline (SCCM/WSUS/Intune/etc.) only updates OS and omits browsers, VPN/VDI clients, or OEM drivers — cite the specific evidence row supporting this.
  • Staleness rule: device shown "online" status but no scan/signature update in ≥30 days = stale, regardless of console status field.
  • EOL detection is independent of scanner output — cross-check OS build/version against known EOL dates even if the vulnerability scanner doesn't flag it.
  • IAM delta must be computed per-account, not aggregated/assumed. Missing EventID or SIEM mapping = flag as "visibility risk," explicitly distinct from a confirmed SLA breach.
  • No inferred compliance. If data is missing, ambiguous, or unverifiable, state so directly: "Insufficient evidence to determine compliance for X" — never round up or assume pass.
Markdown
| Domain & Policy SLA | Preliminary Finding & Root Cause | Affected Hostnames / Accounts (Sampled/Total) | Source Artifact & Sheet Name | Deduction Methodology & Calculated Metrics | |---|---|---|---|---| | Patch Mgmt — ≥95% | Third-party runtime (Chrome, Java) unpatched across fleet; SCCM pipeline covers OS cumulative only | 42/318 hosts (sample: WKS-0231, WKS-0410) | patch_compliance_export.xlsx, Sheet "ThirdPartyApps" | 318 total in-scope; 276 compliant = 86.8% (8.2 pts below 95% SLA) | | IAM — ≥90%, ≤24h | Deactivation delay exceeds 24h SLA for terminated user; legacy forest not revoked | jdoe@domain (term. 2024-03-01 09:00; deactivated 2024-03-04 14:00) | iam_term_log.csv; AD export "LegacyForest_Audit" | Delta = 77h 00m (52h over SLA) |

Guidance:

  • One row per distinct finding (not per host) — use the Affected column for sampling detail, full count in Metrics Breakdown.
  • Always name the exact file and sheet/tab — never "the spreadsheet."
  • Deduction Methodology column must show the arithmetic: total/compliant/percentage, or exact timestamp delta.

Structure as subsections:

A. Fleet Compliance Summary (per domain)

  • Total host count, compliant host count, % compliance, variance from SLA target (e.g., "91.2% vs 95% target = -3.8 pts, SLA BREACH")

B. IAM SLA Breach Detail

  • Per-account table: Account, HR Term Timestamp, Actual Deactivation Timestamp, Delta (hrs/days), Directory/Forest, Status (Compliant / Breach / Visibility Risk — Missing Log)

C. Stale AV Endpoint Inventory

  • Table: Hostname, Last Scan Date, Last DAT/Signature Timestamp, Days Stale, Reported Status (per console)

D. Infrastructure Lifecycle Risks

  • Table: Hostname/System Role, OS Build, EOL Date, Days Past/Until EOL, Scanner-Flagged? (Y/N)

E. Unverified Coverage Register

  • Table: Artifact Filename, Format, Issue (corrupted/unsupported/password-protected/etc.), % of Total Scope Impacted, Resulting Auditable Coverage

Template (adapt language to match input data language):

Subject: [Supplier Name] — Security Evidence Review: Preliminary Findings & Data Validation Request

Dear [Supplier Contact],

As part of our third-party security assessment for [Scope/Site], we have reviewed
the evidence package submitted on [date] against our mandatory compliance baselines:

- Vulnerability Management: ≥95% fleet compliance
- Endpoint Protection (AV/EDR): ≥95% fleet compliance, signatures <7 days
- Patch Management: ≥95% fleet compliance (OS + third-party)
- IAM: ≥90% compliance, deactivation within 24h of termination
- Network & Perimeter: zero unmanaged/unauthorized devices

Our analysis of the submitted artifacts surfaced the following preliminary
observations, which we would like to validate with your team before finalizing
the assessment:

**1. Confirmed Technical Observations**
- [Finding], based on [filename/sheet], sample hosts/accounts: [list]
- ...

**2. Unreadable or Unsupported Submissions**
- [Filename] could not be parsed ([reason]); this segment is currently recorded
  as 0% auditable pending a usable re-export.

These findings are preliminary and pending your validation — we recognize there
may be context, recent remediations, or data formatting issues not reflected in
the raw exports.

To proceed, we kindly request the following within **5 business days**:
1. Uncompressed/re-exported console reports for [specific systems]
2. Forensic log validation (EventID/SIEM trail) for the IAM timestamp deltas noted above
3. Re-export of the corrupted/unreadable archive(s): [filenames]

We appreciate your collaboration in reconciling this data and are happy to
schedule a call to walk through any of these points together.

Best regards,
[Auditor Name]
[Title / Organization]
17 / 20

Example 1: Input: Patch export shows 276/318 compliant per raw rows, but vendor ReadMe claims "97% compliant, fully patched." Output: Matrix row flags the discrepancy itself as a finding — "Vendor-reported 97% not supported by raw data; computed 86.8% based on direct row count" — and the email requests clarification on the calculation method used by the supplier.

Example 2: Input: AV_Console_Export_March.7z fails to extract / is password protected with no key provided. Output: Unverified Coverage Register entry: AV_Console_Export_March.7z — Unverified Coverage due to Unreadable Submission — 100% of AV domain for this cycle — 0% auditable. Email requests a usable re-export within 5 business days, listed separately from confirmed technical gaps.

Recommendation▾
Consider trimming some repetition between the Auditing Rules section and the per-step workflow bullets to tighten length
  • Always separate confirmed gaps (data supports a breach) from visibility gaps (data is missing/unreadable) — never conflate them in tone or in the SLA percentage.
  • Quote exact timestamps and compute deltas explicitly; never say "delayed" without the hour/day figure.
  • Cross-check EOL status against known lifecycle dates even when not prompted by scanner output.
  • When vendor summary and raw data disagree
0
Grade AAI Skill Framework
Scorecard
Criteria Breakdown
Quick Start
14/15
Workflow
15/15
Examples
17/20
Completeness
19/20
Format
14/15
Conciseness
13/15