AI Skill Report Card
Designing Trust Engines
Quick Start15 / 15
When asked to design a trust system, always produce these four artifacts in order:
- Trust Model — entities, trust dimensions, data sources
- Score Logic — formula, weights, decay, thresholds
- Validation Flow — step-by-step validation chain with pass/fail gates
- Risk Report — current risk exposure, edge cases, mitigation
Example minimal output for "driver trust score on a ride-hailing app":
Trust Model:
- Entity: Driver
- Dimensions: Identity Verification, Behavioral History, Peer Rating, Transaction Integrity
- Sources: KYC docs, ride logs, passenger ratings, payment records
Score Logic:
score = 0.30*identity + 0.25*behavior + 0.25*rating + 0.20*transaction
- identity: binary→scaled (verified=100, pending=50, failed=0)
- behavior: rolling 90-day incident-free ratio × 100
- rating: avg passenger rating normalized to 0-100
- transaction: on-time payment ratio × 100
- decay: -2 pts/week if inactive >30 days
- thresholds: 80+ = Trusted, 50-79 = Standard, <50 = Restricted
Validation Flow:
1. Identity check (KYC) → fail = block onboarding
2. Document authenticity scan → fail = manual review
3. Behavioral history pull → flag if incident_count > 3
4. Score computation → assign tier
5. Continuous monitoring → recompute weekly
Risk Report:
- High risk: new accounts with no history (cold start)
- Medium risk: score volatility >15pts/week
- Mitigation: shadow period with capped privileges for new entities
Recommendation▾
Add an example showing a failure case (e.g., a poorly designed score that overweights one signal) to contrast good vs bad outcomes
Workflow14 / 15
Progress checklist for any trust engine design task:
- Identify the entity being trusted (user, driver, merchant, listing, transaction)
- Define trust dimensions (identity, behavior, social proof, financial integrity, tenure)
- Map each dimension to concrete, obtainable data sources
- Design the scoring formula (weighted sum, Bayesian, or decay-based)
- Define score bands/tiers and what privileges/restrictions each tier unlocks
- Design the validation chain (ordered checks, each with pass/fail/escalate outcome)
- Identify cold-start handling (new entities with no history)
- Identify gaming/abuse vectors and countermeasures
- Write the Risk Report summarizing exposure and mitigations
Recommendation▾
Consider trimming some repetition between Quick Start and later sections (Trust Model Design, Score Logic Design) to tighten overall length
Trust Model Design
Structure every Trust Model as:
- Entity: who/what is scored
- Dimensions: 3-5 orthogonal factors (avoid overlapping signals)
- Sources: exact data feeds per dimension — must be things the system can actually access
- Update cadence: real-time, daily, weekly
Common dimension set (reuse unless domain demands otherwise):
- Identity/Verification — KYC, document checks, phone/email verification
- Behavioral History — completed transactions, disputes, violations
- Social/Peer Proof — ratings, reviews, endorsements, referrals
- Financial/Transaction Integrity — payment reliability, chargeback rate
- Tenure/Consistency — account age, activity regularity
Score Logic Design
Default to a weighted linear model unless the domain has strong network effects (then suggest Bayesian/EigenTrust):
score = Σ(weight_i × normalized_signal_i), signals normalized to 0-100
Rules of thumb:
- Weights should sum to 1.0; justify each weight briefly
- Always normalize raw signals to a common 0-100 scale before weighting
- Apply decay for inactivity or aging negative events (so old violations matter less over time)
- Apply caps/floors to prevent single-signal domination (e.g., no dimension >40% impact alone)
- Define at least 3 tiers (e.g., Trusted / Standard / Restricted) with explicit unlocked privileges
- New/cold-start entities get a provisional score (e.g., 50, neutral) with a shadow/probation flag, not 0
Validation Flow Design
Represent as an ordered chain of gates, each with an explicit outcome:
Step N: [Check name]
Pass → continue
Fail → [block / manual review / restrict]
Signal → feeds into Score Logic dimension X
Order checks from cheapest/fastest to most expensive (fail fast on hard blockers like identity fraud before running behavioral analysis).
Risk Report Design
Always include:
- Risk categories: cold-start, score gaming, collusion/sybil, stale scores, data source failure
- Severity ranking: High/Medium/Low with rationale
- Mitigations: concrete countermeasure per risk (probation periods, rate limits, anomaly detection, cross-validation of peer ratings)
- Open gaps: what the current model doesn't cover yet
Examples18 / 20
Example 1: Input: "Design trust scoring for merchants on MusGo marketplace" Output:
Trust Model:
- Entity: Merchant
- Dimensions: Business Verification, Order Fulfillment, Customer Satisfaction, Dispute Rate
- Sources: business license docs, order/shipping logs, review scores, refund/dispute records
Score Logic:
score = 0.25*verification + 0.30*fulfillment + 0.25*satisfaction + 0.20*(100-dispute_rate)
- decay: dispute weight halves every 60 days if no new disputes
- tiers: 85+ Elite Seller, 60-84 Standard, <60 Under Review
Validation Flow:
1. Business license verification → fail = block listing
2. First 10 orders → probation tier, capped visibility
3. Fulfillment rate check (weekly) → <90% = flag for review
4. Dispute audit (per incident) → 3+ unresolved = suspend
Risk Report:
- High: fake reviews inflating satisfaction score → mitigate with verified-purchase-only reviews
- Medium: new merchant cold start → mitigate with probation tier + capped order volume
- Low: seasonal fulfillment dips → mitigate with rolling 90-day window instead of instantaneous rate
Example 2: Input: "How should we handle a user with no history at all?" Output:
Cold-start handling:
- Assign provisional neutral score (50/100), tier = Probation
- Apply reduced privileges (transaction caps, delayed payout, visibility limits)
- Require accelerated verification (phone+email+ID) before any score increase
- Re-evaluate after first 5 completed actions using actual behavioral signals
- Never score cold-start as 0 — that conflates "unknown" with "bad actor"
Recommendation▾
Include guidance on Bayesian/EigenTrust alternative with a concrete formula example, not just a mention
Best Practices
- Keep dimensions orthogonal — don't let two dimensions measure the same underlying signal
- Always show the formula explicitly with real numbers, not just prose
- Tie every score tier to a concrete system behavior (privileges, restrictions, visibility)
- Separate "unknown/new" from "bad" — cold start ≠ distrust
- Make validation flow fail-fast: cheap/hard checks first, expensive/soft checks later
- Always include a decay mechanism so old behavior doesn't permanently define an entity
- Always end with a Risk Report — a trust model without risk analysis is incomplete
Common Pitfalls
- Don't assign new users/entities a score of 0 — treat unknown separately from untrusted
- Don't let one dimension dominate the score (cap max weight/contribution per signal)
- Don't skip normalization — raw signals on different scales break weighted formulas
- Don't design validation flows without explicit fail-path outcomes (every gate needs a defined failure action)
- Don't ignore gaming vectors (fake reviews, sybil accounts, collusion) — always address in Risk Report
- Don't output only the score formula — always deliver all four artifacts (Model, Logic, Flow, Report)