Evaluating API Provider Maturity

Given an API provider's docs/portal/GitHub org, score it like this:

1. Find the apis.yml / apis.json catalog (or equivalent index of resources)
2. Count published API contracts (OpenAPI/AsyncAPI specs)
3. Check for: MCP server, SDKs, CLI, webhooks, idempotency docs, error catalog
4. Score across 7 quality dimensions + agent-readiness checklist (below)
5. Report composite score, flag the weakest 2-3 dimensions, recommend fixes

Example verdict: "Kin Score 77/100 (exemplar), Agent Readiness 53/100 (agent-native). Strong on operational transparency (87) and discoverability (80); weak on governance (0) and access clarity (0) — no well-known catalog, no A2A agent card, no dry-run mode."

Progress:

• Step 1: Inventory the provider's published artifacts
• Step 2: Score the 7 quality dimensions
• Step 3: Score the agent-readiness facets
• Step 4: Apply conditional facets (regulatory posture, create-or-update ergonomics)
• Step 5: Compute composite and write the verdict

Step 1: Inventory artifacts

Look for and list what exists:

• API contracts (OpenAPI count), AsyncAPI specs, JSON-LD contexts, JSON Schema definitions
• SDKs/packages, CLI, MCP server(s)
• Auth docs, rate limits, changelog, status page
• Security posture: security.txt, domain TLS/HSTS/DMARC, trust center, vulnerability disclosure
• Pricing/plans, terms, privacy policy, DPA
• Getting-started guide, API reference, support channel

An absent artifact isn't just "missing docs" — it's a missing scoring input. Treat silence as a zero, not an unknown.

Step 2: Score the 7 quality dimensions (weights are relative, not fixed points)

• Contract Quality — are specs complete, versioned, with real examples (not just schemas)?
• Developer Ergonomics — quickstart time, SDK quality, CLI, sane defaults
• Access Clarity — is auth, scoping, and consent mechanism explicit and machine-readable?
• Operational Transparency — status page, changelog, rate-limit signaling, incident history
• Contract Governance — versioning policy, deprecation process, linting rules (e.g. Spectral) enforced in CI
• Discoverability — is there a machine-readable catalog (apis.json, llms.txt, well-known endpoints)?
• Regulatory Posture (conditional — only applies if tags match a regulated domain like AI/biometrics/health/finance)
• Create-or-Update Ergonomics (conditional — only applies if the API accepts writes): can a caller upsert in one call using a key they already hold, and does the response disclose which branch (create vs update) ran?

Step 3: Score agent-readiness (binary/graded checklist, ~20 facets)

Core (high weight): Machine-Readable Contract, Agentic Access Contract, MCP Server, Machine-Readable Auth, Idempotency, Stable Error Semantics, Rate-Limit Signaling, Typed Event Surface, Agent Skills, Documented Reversibility.

Frontier (usually absent, don't over-penalize if category average is also zero): Well-Known Catalog, Consent & Bot Identity, A2A Agent Card, Dry-Run/Simulate Mode, Delegated User Identity, Protected Resource Metadata, Registration Without a Human, Agentic Commerce Surface.

Step 4: Apply conditional facets

Check tags against regulated-domain list (AI, biometrics, health, finance, etc.) before scoring Regulatory Posture. Check for write endpoints (POST/PUT/PATCH) before scoring Create-or-Update Ergonomics. When a conditional facet applies, it consumes weight from the fixed dimensions — note this explicitly in the report (e.g., "quality facets damped to 75% because two conditional facets carry the other 25%").

Step 5: Compute composite and verdict

Composite = weighted sum of quality dimensions (damped if conditional facets apply). Agent readiness is a separate score, not folded into composite. Label bands: exemplar (≥75), strong (55-74), moderate (35-54), minimal (<35). Always name the top 2 strengths and bottom 2 weaknesses with the specific missing artifact that would fix each.

Example 1: Input: Provider publishes 9 OpenAPI contracts, has a hosted MCP server, TypeScript/Python SDKs, CLI, webhook AsyncAPI spec, JSON-LD context, Spectral ruleset, security.txt, trust center (SOC2/ISO27001), but no well-known catalog, no A2A card, no dry-run mode, no bot-identity/consent mechanism. Output: "Contract Quality strong (multiple versioned specs + Spectral enforcement). Operational Transparency exemplar (status page, changelog, rate limits all documented). Governance weak — Spectral rules exist but no visible deprecation/versioning policy, so Governance scores near zero despite having a linter. Agent readiness is 'agent-native' not 'agent-ready': MCP server and typed events are present, but the frontier facets (well-known catalog, A2A card, dry-run mode, consent/bot-identity) are all absent — these are what separate 'agent-native' from a top-tier agentic score. Recommend: publish a .well-known/apis.json, add a dry-run flag to write endpoints, publish an A2A agent card."

Example 2: Input: Provider has a single OpenAPI file, no SDKs, no changelog, no status page, auth described only in prose. Output: "Composite likely falls in 'minimal' band. Discoverability near zero (no catalog format). Access Clarity near zero (auth not machine-readable). Developer Ergonomics low (no SDK = integration cost high). Before any agent-readiness scoring is meaningful, fix the basics: machine-readable auth scheme declaration and a versioned OpenAPI spec with real request/response examples."