AI Skill Report Card
Designing Quantitative Social Work Research
Quick Start15 / 15
Given a practice/policy problem, produce:
- A quantitative research question with clearly labeled variables
- A hypothesis (null + alternative)
- A recommended design (descriptive, correlational, or experimental)
- A sampling strategy
- Key validity threats and mitigations
Example prompt: "I want to study whether a new case management model reduces client hospital readmissions."
Output:
- Question: Does participation in the [X] case management model (IV) reduce 30-day hospital readmission rates (DV) among adults with chronic illness?
- Hypothesis: H1: Clients receiving the case management model have lower readmission rates than clients receiving standard care. H0: No difference in readmission rates.
- Design: Experimental (RCT) if random assignment is feasible; otherwise quasi-experimental (nonequivalent control group).
- Sampling: Random assignment within a purposive sample of eligible clients at intake; power analysis to determine sample size.
- Validity: Control for selection bias (randomization), attrition (track dropouts), history/maturation (control group), and measurement reliability (validated readmission tracking).
Recommendation▾
Add an example showing a purely descriptive design (all current examples lean correlational/experimental) for fuller design-type coverage
Workflow15 / 15
Progress:
- Step 1: Clarify the practice problem and convert it into a measurable research question
- Step 2: Identify dependent (outcome) and independent (predictor) variables; define operationally
- Step 3: Write testable hypotheses (null and alternative)
- Step 4: Select a design based on the question's purpose
- Step 5: Choose a sampling method appropriate to the design and population
- Step 6: Identify measurement instruments and assess reliability/validity
- Step 7: Anticipate threats to internal and external validity and propose mitigations
- Step 8: Note appropriate analysis approach (descriptive stats, correlation, regression, group comparison) — match to design, not perform full stats
Step 1: Convert problem into a research question
Ask: What is being measured? In whom? Under what conditions? Quantitative questions should be answerable with numbers, not narratives. Avoid "why/how does it feel" phrasing — reserve for qualitative work.
Step 2: Define variables
- Dependent variable (DV): the outcome of interest (e.g., depression score, recidivism rate)
- Independent variable (IV): the presumed cause/predictor (e.g., program participation, demographic factor)
- State the level of measurement for each (nominal, ordinal, interval, ratio) — this determines which statistical tests are later usable.
Step 3: Write hypotheses
Frame as directional or non-directional. Always pair a null (H0: no relationship/difference) with an alternative (H1).
Step 4: Select design — decision guide
| Purpose | Design | Notes |
|---|---|---|
| Describe characteristics/prevalence, no comparison | Descriptive | Surveys, needs assessments; no manipulation of variables |
| Examine relationship/association between variables, no causal claim | Correlational | Use when random assignment isn't ethical/feasible; reports correlation coefficients (e.g., Pearson's r) |
| Test cause-and-effect, requires manipulation of IV | Experimental | Requires random assignment + control group for strongest causal inference |
| Cause-effect desired but randomization not possible | Quasi-experimental | Nonequivalent groups, pre/post designs; weaker internal validity, common in field settings |
Step 5: Sampling method
- Probability sampling (random, stratified, cluster, systematic) — needed for generalizability/external validity; required for true experiments.
- Non-probability sampling (convenience, purposive, snowball) — common in social work practice research where access is limited; weakens external validity, note as a limitation.
- Always report intended sample size and, when possible, justify via power analysis.
Step 6: Measurement
- Identify existing validated instruments (scales) before creating new ones.
- Report reliability (e.g., Cronbach's alpha) and validity (construct, content, criterion) evidence.
Step 7: Validity threats
Internal validity (does the IV actually cause the DV change?):
- History, maturation, testing effects, instrumentation, statistical regression, selection bias, attrition/mortality, diffusion of treatment
External validity (do results generalize?):
- Sampling bias, unrepresentative settings, reactivity to being studied (Hawthorne effect)
Mitigations: randomization, control groups, blinding, standardized protocols, tracking attrition, using representative samples.
Step 8: Match analysis to design
- Descriptive → means, frequencies, standard deviations
- Correlational → Pearson's r / Spearman's rho, regression
- Experimental/quasi-experimental → t-tests, ANOVA, regression with control variables
- Note parametric vs. non-parametric test choice depends on data distribution and measurement level
Recommendation▾
Consider a brief troubleshooting note on what to do when power analysis isn't feasible (small agency samples) since this is a common real-world constraint
Examples18 / 20
Example 1: Input: "I want to know if social workers with more years of experience report less burnout." Output:
- Question: Is there a relationship between years of professional experience (IV, ratio) and reported burnout level (DV, interval, via validated burnout scale) among licensed social workers?
- Hypothesis: H1: Years of experience is negatively correlated with burnout score. H0: No correlation exists.
- Design: Correlational (no manipulation possible; both variables naturally occurring)
- Sampling: Stratified random sample of licensed social workers by years-of-experience bracket, recruited via state licensing board list
- Analysis: Pearson's r; consider regression to control for confounds (caseload size, agency type)
- Validity note: Self-report burnout scale subject to social desirability bias; cross-sectional design cannot establish causality
Example 2: Input: "We're piloting a new group intervention for youth anxiety and want to know if it works." Output:
- Question: Does participation in the [Y] group intervention (IV) reduce anxiety symptoms (DV, measured via GAD-7) among adolescents aged 13-17 compared to a waitlist control?
- Hypothesis: H1: Intervention group shows greater reduction in GAD-7 scores than waitlist control at post-test. H0: No difference between groups.
- Design: Experimental (RCT with waitlist control) if random assignment feasible; otherwise quasi-experimental pre/post nonequivalent groups
- Sampling: Random assignment from a pool of referred youth meeting eligibility criteria; calculate needed sample size via power analysis for detecting moderate effect size
- Validity threats: Maturation (adolescents' anxiety may change naturally), attrition (track dropout by group), testing effects (repeated GAD-7 administration), Hawthorne effect
- Analysis: Independent-samples t-test or ANCOVA controlling for baseline scores
Recommendation▾
The design decision table is strong; could add a similar quick-reference table for choosing sampling methods to mirror that clarity
Best Practices
- Start with the question's purpose (describe, relate, or cause) — this single decision determines design, sampling rigor needed, and analysis options.
- Operationally define every variable before collecting data; vague constructs ("wellbeing," "success") must be tied to a specific instrument.
- Prefer existing validated instruments over researcher-created ones; report their reliability/validity evidence.
- Always pair internal validity discussion (causal confidence) with external validity discussion (generalizability) — a strong RCT with a convenience sample still has limited generalizability.
- When randomization isn't possible (common in real-world agency settings), be explicit that the design is quasi-experimental and name the specific internal validity threats this introduces.
- Match statistical test to measurement level and distribution (parametric vs. non-parametric) rather than defaulting to one test.
Common Pitfalls
- Labeling a study "experimental" without random assignment and a control group — this is quasi-experimental at best.
- Claiming causation from correlational data ("increases," "causes," "leads to") — use "associated with" instead.
- Skipping operational definitions and measurement instrument selection, leading to unmeasurable constructs.
- Ignoring attrition/dropout tracking, which silently undermines internal validity in longitudinal or intervention designs.
- Using convenience samples while claiming generalizable/population-level findings without caveating external validity limits.
- Choosing statistical tests without checking the data's level of measurement or distribution shape.