AI Skill Report Card
Designing Registered Report Protocols
Quick Start14 / 15
Given a conceptual model (e.g., "prompts have three components: X, Y, Z") and a research question, produce a Stage 1 protocol with this skeleton:
- Introduction: state the gap, name the conceptual model, list the hypotheses in one sentence each.
- Background: position the model against related work; state explicitly what nobody has done yet ("Research Gap" paragraph).
- Hypotheses: 3–5 numbered, falsifiable, each traceable to a specific analysis/statistical test.
- Study Overview: describe data sources and their roles (confirmatory vs. replication vs. exploratory) and a timeline figure.
- Data Collection: one subsection per method, each with recruitment/sampling, design, and power analysis.
- Measures/Coding: operationalize every construct in the hypotheses with a coding scheme and reliability target.
- Analysis Plan: map each hypothesis to a specific statistical test, decided before seeing data.
- Threats to Validity: internal, external, construct, conclusion — tied back to design choices already made.
Recommendation▾
Example 2 is weak—showing a placeholder response for gibberish input doesn't demonstrate real methodology; replace with a second substantive example (e.g., a different domain like testing/CI or requirements engineering) to show breadth.
Workflow14 / 15
Progress:
- Extract the conceptual model into named, orthogonal components
- Draft one falsifiable hypothesis per research question, each stating direction and mechanism
- Identify which hypothesis is the "key" one (usually the interaction/temporal one) vs. supporting/validating ones
- Design triangulated data collection: one method for causal claims, one for scale/generalizability, one exploratory
- Assign explicit roles to each data source per hypothesis (confirmatory / co-primary replication / exploratory-only) — never let one dataset silently serve two incompatible roles
- Run power analysis for every planned inferential test; report software, parameters, and resulting minimum N
- Define coding scheme with dual-coder reliability threshold (κ ≥ 0.75 is a common bar) and state what happens if it's not met — this should itself be treated as disconfirming evidence for the underlying model
- Build a project timeline diagram tied to milestones (design → pilot → collection → annotation → analysis → submission)
- Write threats-to-validity section anticipating reviewer objections to each design choice
- Cross-check: every hypothesis has a data source, a measure, a statistical test, and a stated minimum sample size
Recommendation▾
Show the actual power analysis output/format (e.g., 'G*Power: f=0.25, alpha=.05, power=.80 → N=128') rather than just instructing to report it, to make the guidance more concrete and actionable.
Examples15 / 20
Example 1: Input: "We have a model that splits code review comments into Style, Logic, and Architecture concerns. We want to know if reviewer seniority changes the mix, and if the mix predicts whether a PR gets merged." Output:
- H1 (Seniority Effect): The proportion of Architecture-focused comments increases with reviewer seniority (years of experience), tested via regression with seniority as continuous predictor.
- H2 (Mix Predicts Merge): PRs with higher proportion of Logic comments relative to Style comments are more likely to be merged without further revision, tested via logistic regression.
- Data collection: controlled experiment with recruited reviewers (causal, small n) + mined GitHub PR review data (observational, large n, exploratory role for H2 generalizability).
- Coding scheme: two independent coders label each comment into the three categories; report Cohen's κ; if κ<0.75, treat the taxonomy itself as needing revision before hypothesis testing proceeds.
Example 2: Input: "gibberish nonsense asdkjfh" Output: A minimal placeholder skill only — see below; this expertise area requires real research content to operate on.
Recommendation▾
Add a brief example of a filled-out threats-to-validity section or coding scheme snippet, since these are named as required components but never demonstrated in full.
Best Practices
- State hypotheses as directional predictions with mechanisms, not vague associations: "X increases relative to Y because Z," not "X is related to Y."
- Distinguish the "setup" hypotheses from the "key" hypothesis. Descriptive/validating hypotheses (e.g., component evolves over time) exist to support the interaction/temporal hypothesis that is the actual contribution.
- Give every data source an explicit epistemic role. Don't let a single dataset be asked to support causal claims it can't bear — controlled experiments get causal/confirmatory weight, community/mined data get replication or exploratory weight only.
- Fold construct validation into the design. If the underlying conceptual model can fail to be operationalized (e.g., can't be reliably coded), state this as a possible outcome and treat it as evidence, not as a nuisance to be explained away.
- Compute power analysis for the actual test you'll run (e.g., repeated-measures ANOVA with specific effect size, alpha, correlation among measures), not a generic rule of thumb, and report the software/parameters used.
- Tie recruitment strata to variables used in analysis. Categorical recruitment buckets (e.g., junior/senior) should not be the analysis variable if a continuous version (years of experience) is available — use continuous variables in models, categorical only for stratified sampling.
- Make the timeline concrete with named milestones and explicit role assignments (e.g., "Master's student contributes to annotation").
- Write the paper before the data exists. Every methodological choice must be locked in the Stage 1 protocol; nothing in Results should introduce a new hypothesis or measure.
Common Pitfalls
- Writing hypotheses that aren't falsifiable or don't specify direction/mechanism.
- Letting one dataset implicitly serve as both exploratory source and confirmatory test for the same hypothesis — this collapses the value of triangulation.
- Skipping power analysis or using a generic "N=30 is enough" justification instead of parameterized calculation per test.
- Treating the conceptual model as unfalsifiable — always include a condition under which the model itself would be considered disconfirmed (e.g., failed inter-rater reliability, degenerate distributions).
- Conflating recruitment strata with analysis variables (using junior/mid/senior as a regression predictor when continuous experience data was collected).
- Omitting the "what if this fails" branch for coding/annotation reliability.
- Overloading the introduction with related work instead of a crisp gap statement — save depth for the Background section.