Auditing Third Party Cyber Risk Evidence
Auditing Third-Party Cyber Risk Evidence
Given supplier name, scope, and raw evidence artifacts, produce:
- Evidence & Findings Traceability Matrix (Markdown table)
- Granular Metrics Breakdown (counts, percentages, deltas, EOL risks, unverified coverage)
- Supplier Verification Email (collaborative, evidence-grounded, in the input's language)
Never summarize vendor-provided summaries. Parse raw rows/log lines directly. If any archive is unreadable, mark it "Unverified Coverage due to Unreadable Submission" and count 0% for that segment.
| Domain | SLA Threshold | Key Sub-Check |
|---|---|---|
| Vulnerability Management | ≥95% fleet compliance | Unresolved Critical (9.0–10.0) / High (7.0–8.9) ≤5% of in-scope fleet |
| Endpoint Protection (AV/EDR) | ≥95% fleet compliance | Agent health, signature age <7 days, stale/offline tracking |
| Patch Management | ≥95% fleet compliance | OS cumulative updates AND third-party runtimes/apps |
| IAM (physical + logical) | ≥90% compliance | Deactivation ≤24h from HR termination effective time, all domains/directories |
| Network & Perimeter | 0 tolerance | No unmanaged/unauthorized/consumer-grade hardware in production or Secure Workspace |
If custom thresholds are supplied in the input, use those instead and state explicitly which baseline set is in effect.
Progress:
- Step 1: Intake scope — confirm Supplier Name, Scope/Site/Geography, and inventory every submitted artifact (filename, format, sheet names)
- Step 2: Triage artifacts — flag unreadable/corrupted/unsupported files immediately; do not attempt to infer their contents
- Step 3: Parse vulnerability management evidence — compute fleet %, isolate Critical/High unresolved findings, identify root cause (scan scope gaps, agent failures)
- Step 4: Parse endpoint protection evidence — compute agent health %, flag signatures >7 days old, flag devices reported online with no scan/update in 30 days as "stale"
- Step 5: Parse patch management evidence — separate OS cumulative patch compliance from third-party app/runtime/OEM driver patch compliance; identify pipeline root cause (e.g., SCCM covering OS only)
- Step 6: Parse IAM evidence — for every termination record, compute [Deactivation Timestamp] − [Effective HR Termination Timestamp] in hours/days; check all directories/forests for simultaneity; flag missing log trails as visibility risks, not compliance
- Step 7: Parse network/perimeter evidence — identify any unmanaged, unauthorized, or consumer-grade devices in scope
- Step 8: Identify infrastructure lifecycle risks — flag EOL/near-EOL OS on critical infrastructure (DCs, DNS, etc.) independent of scanner flags
- Step 9: Build the Evidence & Findings Traceability Matrix — one row per distinct finding, cite exact source file/sheet
- Step 10: Build the Granular Metrics Breakdown
- Step 11: Draft the Supplier Verification Email — same language as input data, collaborative tone, 5-business-day ask
- Step 12: State all assumptions and data gaps explicitly; never infer compliance from absence of data
- Deep artifact parsing only. Ignore vendor ReadMe/summary claims. Cross-reference raw rows against stated percentages — if a vendor summary says 97% but raw data supports 91%, report the computed 91% and note the discrepancy as a finding itself.
- Unreadable archives → 0% auditable for that coverage segment. Label exactly:
"Unverified Coverage due to Unreadable Submission". - Root-cause, not just symptom. For patch gaps, explicitly state whether the pipeline (SCCM/WSUS/Intune/etc.) only updates OS and omits browsers, VPN/VDI clients, or OEM drivers — cite the specific evidence row supporting this.
- Staleness rule: device shown "online" status but no scan/signature update in ≥30 days = stale, regardless of console status field.
- EOL detection is independent of scanner output — cross-check OS build/version against known EOL dates even if the vulnerability scanner doesn't flag it.
- IAM delta must be computed per-account, not aggregated/assumed. Missing EventID or SIEM mapping = flag as "visibility risk," explicitly distinct from a confirmed SLA breach.
- No inferred compliance. If data is missing, ambiguous, or unverifiable, state so directly: "Insufficient evidence to determine compliance for X" — never round up or assume pass.
Markdown| Domain & Policy SLA | Preliminary Finding & Root Cause | Affected Hostnames / Accounts (Sampled/Total) | Source Artifact & Sheet Name | Deduction Methodology & Calculated Metrics | |---|---|---|---|---| | Patch Mgmt — ≥95% | Third-party runtime (Chrome, Java) unpatched across fleet; SCCM pipeline covers OS cumulative only | 42/318 hosts (sample: WKS-0231, WKS-0410) | patch_compliance_export.xlsx, Sheet "ThirdPartyApps" | 318 total in-scope; 276 compliant = 86.8% (8.2 pts below 95% SLA) | | IAM — ≥90%, ≤24h | Deactivation delay exceeds 24h SLA for terminated user; legacy forest not revoked | jdoe@domain (term. 2024-03-01 09:00; deactivated 2024-03-04 14:00) | iam_term_log.csv; AD export "LegacyForest_Audit" | Delta = 77h 00m (52h over SLA) |
Guidance:
- One row per distinct finding (not per host) — use the Affected column for sampling detail, full count in Metrics Breakdown.
- Always name the exact file and sheet/tab — never "the spreadsheet."
- Deduction Methodology column must show the arithmetic: total/compliant/percentage, or exact timestamp delta.
Structure as subsections:
A. Fleet Compliance Summary (per domain)
- Total host count, compliant host count, % compliance, variance from SLA target (e.g., "91.2% vs 95% target = -3.8 pts, SLA BREACH")
B. IAM SLA Breach Detail
- Per-account table: Account, HR Term Timestamp, Actual Deactivation Timestamp, Delta (hrs/days), Directory/Forest, Status (Compliant / Breach / Visibility Risk — Missing Log)
C. Stale AV Endpoint Inventory
- Table: Hostname, Last Scan Date, Last DAT/Signature Timestamp, Days Stale, Reported Status (per console)
D. Infrastructure Lifecycle Risks
- Table: Hostname/System Role, OS Build, EOL Date, Days Past/Until EOL, Scanner-Flagged? (Y/N)
E. Unverified Coverage Register
- Table: Artifact Filename, Format, Issue (corrupted/unsupported/password-protected/etc.), % of Total Scope Impacted, Resulting Auditable Coverage
Template (adapt language to match input data language):
Subject: [Supplier Name] — Security Evidence Review: Preliminary Findings & Data Validation Request
Dear [Supplier Contact],
As part of our third-party security assessment for [Scope/Site], we have reviewed
the evidence package submitted on [date] against our mandatory compliance baselines:
- Vulnerability Management: ≥95% fleet compliance
- Endpoint Protection (AV/EDR): ≥95% fleet compliance, signatures <7 days
- Patch Management: ≥95% fleet compliance (OS + third-party)
- IAM: ≥90% compliance, deactivation within 24h of termination
- Network & Perimeter: zero unmanaged/unauthorized devices
Our analysis of the submitted artifacts surfaced the following preliminary
observations, which we would like to validate with your team before finalizing
the assessment:
**1. Confirmed Technical Observations**
- [Finding], based on [filename/sheet], sample hosts/accounts: [list]
- ...
**2. Unreadable or Unsupported Submissions**
- [Filename] could not be parsed ([reason]); this segment is currently recorded
as 0% auditable pending a usable re-export.
These findings are preliminary and pending your validation — we recognize there
may be context, recent remediations, or data formatting issues not reflected in
the raw exports.
To proceed, we kindly request the following within **5 business days**:
1. Uncompressed/re-exported console reports for [specific systems]
2. Forensic log validation (EventID/SIEM trail) for the IAM timestamp deltas noted above
3. Re-export of the corrupted/unreadable archive(s): [filenames]
We appreciate your collaboration in reconciling this data and are happy to
schedule a call to walk through any of these points together.
Best regards,
[Auditor Name]
[Title / Organization]
Example 1: Input: Patch export shows 276/318 compliant per raw rows, but vendor ReadMe claims "97% compliant, fully patched." Output: Matrix row flags the discrepancy itself as a finding — "Vendor-reported 97% not supported by raw data; computed 86.8% based on direct row count" — and the email requests clarification on the calculation method used by the supplier.
Example 2:
Input: AV_Console_Export_March.7z fails to extract / is password protected with no key provided.
Output: Unverified Coverage Register entry: AV_Console_Export_March.7z — Unverified Coverage due to Unreadable Submission — 100% of AV domain for this cycle — 0% auditable. Email requests a usable re-export within 5 business days, listed separately from confirmed technical gaps.
- Always separate confirmed gaps (data supports a breach) from visibility gaps (data is missing/unreadable) — never conflate them in tone or in the SLA percentage.
- Quote exact timestamps and compute deltas explicitly; never say "delayed" without the hour/day figure.
- Cross-check EOL status against known lifecycle dates even when not prompted by scanner output.
- When vendor summary and raw data disagree