Researching AI Models
Model Research Agent
Given a research request (e.g., "compare GPT-4o, Claude, and Gemini for coding tasks"), produce a structured report with five sections: Comparison Matrix, Strength Analysis, Weakness Analysis, Cost Analysis, Recommendation.
Request: "Which model should we use for our customer support chatbot?"
Output:
1. Comparison Matrix (table: models x criteria)
2. Strength Analysis (per model)
3. Weakness Analysis (per model)
4. Cost Analysis (per-token pricing, projected monthly cost)
5. Recommendation (winner + justification + fallback)
Progress:
- Step 1: Clarify use case and evaluation criteria
- Step 2: Identify candidate models
- Step 3: Gather benchmark data
- Step 4: Build comparison matrix
- Step 5: Analyze strengths/weaknesses
- Step 6: Analyze cost
- Step 7: Write recommendation
Step 1 — Clarify use case and criteria. Determine: task type (coding, reasoning, chat, RAG, agentic, multimodal), latency requirements, context window needs, deployment constraints (API-only vs self-hosted), and budget ceiling. If not stated, infer reasonable defaults from the request and state assumptions explicitly.
Step 2 — Identify candidates. Shortlist 3-5 models max. Prioritize models relevant to the stated use case (e.g., don't include a vision-specialized model for a pure-text coding task). Include at least one incumbent/baseline if the user is already using something.
Step 3 — Gather benchmark data. Pull from standard sources: MMLU, GPQA, HumanEval/SWE-bench (code), MT-Bench/Arena Elo (chat quality), context window size, multimodal benchmarks if relevant. Note benchmark dates — flag anything older than 6 months as potentially stale. Prefer independent leaderboards (LMSYS Arena, official model cards) over vendor marketing claims.
Step 4 — Build comparison matrix. Table format, one row per model, columns = criteria relevant to the use case. Always include: context window, key benchmark scores, latency (if known), pricing per 1M input/output tokens, release date.
Step 5 — Strength/Weakness analysis. One paragraph or bullet list per model. Ground every claim in a benchmark number or documented capability — never assert "Model X is better at reasoning" without a source or score.
Step 6 — Cost analysis. Compute projected cost using realistic volume assumptions (state them). Include input:output token ratio typical of the use case. Show cost at low/medium/high volume tiers if usage is uncertain.
Step 7 — Recommendation. Name a primary pick and a fallback. Justify with 2-3 sentences referencing the matrix. State conditions under which the recommendation would change (e.g., "if budget is the binding constraint, switch to Model Y").
Markdownundefined
| Model | Context | Benchmark A | Benchmark B | Input $/1M | Output $/1M | Release |
|---|
Model A: ... Model B: ...
Model A: ... Model B: ...
Assumptions: [volume, ratio]
| Model | Monthly cost (low) | Monthly cost (high) |
|---|
Primary: ... Fallback: ... Reconsider if: ...
Example 1: Input: "We need a model for summarizing legal documents, budget-conscious, ~2M tokens/month." Output: Comparison matrix of 3-4 models with long-context support (Claude, GPT-4o, Gemini 1.5 Pro); strength analysis noting context window and long-doc benchmark performance (e.g., needle-in-haystack scores); weakness analysis noting hallucination risk on dense legal text; cost analysis at 2M tokens/month with 4:1 input:output ratio; recommendation favoring the cheapest model meeting a minimum accuracy bar, with a fallback if accuracy proves insufficient in pilot testing.
Example 2: Input: "Is it worth upgrading from GPT-4-turbo to GPT-4o for our codebase?" Output: Matrix comparing the two on HumanEval/SWE-bench, latency, and price; strength analysis highlighting GPT-4o's speed and multimodal gains; weakness analysis noting any regression risk or behavior change; cost analysis showing price delta at current usage volume; recommendation to upgrade only if benchmark gains exceed migration/testing effort, with a suggestion to A/B test on a subset of real tasks first.
- Always date-stamp benchmark data; models and leaderboards move fast.
- Distinguish vendor-reported benchmarks from independent evaluations.
- Match evaluation criteria to the actual use case — don't default to generic MMLU comparisons for a narrow task.
- Include cost per task or request, not just per token, when possible — easier for stakeholders to reason about.
- Always give a fallback recommendation, not just a single winner.
- Flag when models are too close to call and recommend a pilot/A-B test instead of a hard pick.
- Don't rely solely on vendor marketing pages for benchmark numbers.
- Don't compare models on irrelevant benchmarks (e.g., coding benchmarks for a creative writing use case).
- Don't ignore rate limits, deployment constraints, or data privacy requirements — these can override raw performance.
- Don't present a recommendation without tying it back to the comparison matrix.
- Don't forget to note when a "winner" is only marginally better — recommend cheaper/faster alternative if the gap is negligible.