AI Skill Report Card
Managing Sentry Bug Fix Orchestration
Orchestrating Sentry Bug Fixes
Quick Start10 / 15
Bash# Tier 1: Full automation (≥94% confidence) @sentry-fix-orchestrator https://sentry.io/organizations/acme/issues/4521789/ # Session persistence mkdir evidence/4521789/ echo '{"issueId":"4521789","tier":1,"confidence":96,"currentStep":"2A.3"}' > evidence/4521789/session.json
Recommendation▾
Replace abstract descriptions with concrete input/output examples - show actual Sentry URLs, real error messages, and specific code fixes rather than placeholder examples
Workflow12 / 15
Core Pipeline:
- Parse & Fetch: Extract issue ID, fetch via Sentry API, read stack trace files
- Analyze: WHY pyramid (5 levels), value lifecycle tracing, confidence scoring
- Tier Classification: ≥94%=T1(auto), 70-84%=T2(assisted), <70%=T3(analysis)
- Replicate: Unit→Integration→E2E cascade with 3-5 scenarios
- Fix & Validate: Generate fix, validate against evidence, create PR
Gate System:
JavaScript// Gate #3: Replication Proof (required before fix generation) const validProofs = [ 'replication-result.json:status===REPRODUCED', 'zero-execution-evidence.json:hasReplay===true', 'session.json:userApproval===FIX_WITHOUT_REPRO' ];
Recommendation▾
Remove over-explanation of basic concepts like 'parse URL' and 'fetch via API' that Claude already understands - focus on the unique orchestration methodology
Examples8 / 20
Example 1: Tier 1 Full Auto
Input: https://sentry.io/acme/issues/4521789/events/abc123/
Output:
TIER 1 | COMPLETE | PR_CREATED | 96%→T1 | session.json:COMPLETE | none
Root Cause: UserService.getProfile() receives null userId from AuthMiddleware
Fix: Add null check in AuthMiddleware.extractUserId() line 47
Files: auth/middleware.js (+3 lines), tests/auth.test.js (+15 lines)
PR: #1847 BUG(BOSIGN-4521789): Null userId crashes user profile endpoint
Evidence: evidence/4521789/replication-result.json (REPRODUCED via unit test)
Example 2: Tier 2 Assisted Input: Database connection timeout in checkout flow Output:
TIER 2 | USER_VALIDATION | AWAITING_CONFIRMATION | 78%→T2 | session.json:SCENARIOS_GENERATED | none
Top Hypothesis (78%): Connection pool exhaustion during high concurrency
Test Scenario A: Simulate 50 concurrent checkout requests
Test Scenario B: Reduce pool size to 5, trigger with 10 requests
Test Scenario C: Monitor connection lifecycle during payment processing
Confirm which scenario to run? Or describe what you observed?
Example 3: Tier 3 Analysis Input: Intermittent React component crash Output:
TIER 3 | ANALYSIS_COMPLETE | ENGINEER_REVIEW | 45%→T3 | session.json:ESCALATED | insufficient_repro_data
Analysis Summary:
- Error: Cannot read property 'name' of undefined in UserCard.jsx:23
- Hypotheses: State race condition (30%), Async data loading (25%), Props validation (20%)
- Missing: User interaction flow, network timing, component lifecycle events
- Recommendation: Add error boundaries, implement loading states, trace prop flow
Recommendation▾
Provide actual templates for session.json structure, confidence calculation formulas, and tier classification criteria rather than just describing them
Best Practices
Confidence Calculation:
JavaScript// Base confidence from evidence quality let confidence = 0; if (stackTraceComplete) confidence += 20; if (breadcrumbsAvailable) confidence += 15; if (userActionsKnown) confidence += 15; // WHY pyramid depth bonus confidence += whyLevels * 10; // max +50 // Scenario validation if (scenariosConfirmed >= 2) confidence += 20; if (allScenariosEliminated) confidence = Math.min(94, confidence);
Replication Strategy:
Bash# Unit test (fastest) jest tests/auth.test.js --testNamePattern="null userId" # Integration test (medium) npm test -- --grep "auth middleware integration" # E2E test (slowest, fallback only) npx playwright test checkout-flow.spec.js
Session State Management:
JSON{ "issueId": "4521789", "tier": 1, "confidence": 96, "currentStep": "2A.6", "gates": { "dataAcquisition": true, "analysis": true, "replicationProof": true, "fixQuality": true }, "evidencePath": "evidence/4521789/", "fixProposal": { "files": ["auth/middleware.js"], "confidence": 96, "method": "DIRECT" } }
Common Pitfalls
- Gate bypassing: Never skip replication proof even at 99% confidence - generate-fix is FORBIDDEN without valid evidence files
- Tier inflation: Don't round 84.9% to Tier 1 - use strict thresholds (≥94%, ≥70%, <70%)
- Pattern matching confusion: PATTERN_MATCH ≠ REPRODUCED - only REPRODUCED status unlocks fix generation
- User confirmation shortcuts: User saying "yes, that's the issue" ≠ replication proof - still need test execution
- Evidence assumptions: Physically read evidence files - don't infer existence from previous steps