Analyzing Developer Needs for AI Tooling
Given a corpus of open-ended survey responses about a domain (e.g., "where do you want AI support" and "where do you not want AI to help"), produce for each identified need:
- The system: a concrete, buildable AI capability
- The problem it solves: grounded in respondent language, not abstraction
- What makes it hard to build: technical/organizational obstacles
- The constraints on its behavior: explicit boundaries respondents impose
Example output entry:
System: Pre-merge quality signal injector — Developers want AI to flag likely defects, missing tests, and security gaps at authorship time, not at review time. Hard to build because it requires calibrated confidence without false-positive fatigue. Constraint: must surface uncertainty explicitly and never auto-merge or auto-approve.
Progress:
- Step 1: Define a grounded task taxonomy for the domain
- Step 2: Design paired opportunity/constraint elicitation questions per task category
- Step 3: Collect and clean responses (filter incomplete, patterned, attention-check failures)
- Step 4: Run multi-model council-based theme discovery per response track
- Step 5: Reconcile themes across models with human-in-the-loop adjudication
- Step 6: Synthesize each theme into a "system" with problem, difficulty, constraints
- Step 7: Look for cross-cutting patterns (guardrails, boundaries, identity signals)
- Step 8: Name the overarching pattern/tension the data reveals
Step 1: Grounded Task Taxonomy
Don't invent categories from theory alone. Triangulate from:
- Existing work-diary/time-use studies of the target population
- Prior surveys/adoption studies in the same domain
- Pilot sessions with practitioners outside the research team
Step 2: Paired Elicitation Design
For each task category, ask two complementary open-ended questions:
- Opportunity: "Where do you want AI to play the biggest role in [category]?"
- Constraint: "What aspects do you not want AI to handle in [category], and why?"
This pairing surfaces both desired capability and boundary conditions in the same breath — critical, because boundaries are often not the inverse of capabilities (people refuse things AI could plausibly do well).
Step 3: Data Cleaning
Exclude: incomplete responses, patterned/straight-lined responses, attention-check failures, respondents with no relevant experience. Report exclusion counts transparently.
Step 4: Multi-Model Council Theme Discovery
Run 2-3 independent frontier models over the same response track, each performing open coding independently (avoids single-model idiosyncrasy/bias). Use identical prompts and track-specific codebooks for opportunity vs. constraint tracks.
Step 5: Reconciliation
A reconciliation step merges/adjudicates themes across model outputs. Keep a human in the loop at this stage — automated agreement metrics are insufficient for interpretive nuance; a researcher resolves disagreements and validates edge cases.
Step 6: System Synthesis
For every validated theme, write it up as a discrete "system" using this fixed template:
- Name (concrete, buildable, not aspirational)
- Problem solved (in respondent language)
- What makes it hard (technical, organizational, or trust obstacles)
- Constraints developers place on it (explicit boundaries)
Step 7: Cross-Cutting Pattern Extraction
After synthesizing individual systems, step back and ask:
- What guardrails recur across most/all systems? (e.g., authority scoping, provenance, uncertainty signaling, least-privilege access)
- Where do "opportunity" and "constraint" responses cluster on the same task — and what does that co-occurrence reveal?
Step 8: Name the Tension
Give the overarching pattern a memorable name (e.g., "bounded delegation") that captures why the boundary exists — usually tied to professional identity, accountability, or ownership, not capability limits.
Example 1: Input: Developer responses across "Quality & Risk Management" opportunity/constraint questions in a large survey. Output: System — "Pre-commit compliance and security linting with explainable rationale." Problem: developers want risk signals earlier, before code reaches review, to offset accelerating AI-generated code volume. Hard to build: requires domain-specific compliance rules that don't produce alert fatigue. Constraints: must show provenance of the rule triggered, must not auto-block commits without human override, must flag confidence level.
Example 2: Input: Responses in "Meta-work" category about documentation. Output: System — "Living documentation synchronizer" that detects code-doc drift and proposes (not auto-commits) updates. Constraint: developers refuse full automation of documentation authorship even though they want drift detection — because authored docs are a form of accountability and shared understanding, not just information transfer.
- Ground every system in respondent quotes/language, not researcher inference — traceability strengthens validity.
- Always pair opportunity and constraint analysis on the same task category — analyzing them separately loses the boundary-drawing signal.
- Use multiple independent models for theme discovery, then reconcile with a human — single-pass LLM coding is prone to premature convergence.
- Distinguish "hard to build" (technical) from "constrained" (normative) — conflating them misrepresents why a boundary exists.
- Look for boundaries that persist despite acknowledged AI capability — these are identity/accountability boundaries, not capability gaps, and are unlikely to shift as models improve.
- Report exclusion criteria and counts transparently for reproducibility.
- Synthesize a memorable name for the overarching tension — this becomes the paper's/analysis's conceptual contribution, not just the list of systems.
- Don't let a single model's thematic output stand unreconciled — cross-model disagreement is signal, not noise.
- Don't assume constraints are simply "the opposite of adoption" — developers often refuse automation for tasks they admit AI could handle competently.
- Don't produce vague system names ("better testing AI") — always specify what triggers it, what it outputs, and what it must never do autonomously.
- Don't skip the cross-cutting guardrail analysis — the individual systems are less valuable than the recurring constraint patterns across them.
- Don't treat this as a productivity/adoption study — the goal is prescriptive (what to build) and boundary-defining (what not to automate), not descriptive of current usage.