Evaluating Vision Language Procedural Reasoning

To evaluate whether a vision-language model can truly "do" rather than just "describe," build a benchmark around a domain with a formal, compilable symbolic grammar (DSL, code, CAD script, recipe format). This lets you validate outputs by execution, not just text similarity.

Minimal task ladder:

1. Recognition — classify components from an image (multi-label)
2. Comprehension — match image to correct instruction among distractors (MCQ)
3. Generation — produce free-form natural-language instructions from an image
4. Formalization — translate instructions/image into an executable DSL program, validated by compilation

Python
metrics = {
    "task_a_recognition": ["precision", "recall", "f1"],
    "task_b_comprehension": ["accuracy"],
    "task_c_generation": ["BLEU", "ROUGE-L", "ChrF"],
    "task_d_formalization": ["valid_pattern_rate", "structural_similarity"]
}

The key insight: track how performance decays as you move from surface-level metrics (BLEU) to execution-based metrics (compiles + structurally correct). That gap is the finding.

Progress:
- [ ] Step 1: Identify a domain with formal executable grammar
- [ ] Step 2: Build/curate large-scale paired data (image + structured procedure)
- [ ] Step 3: Validate data extraction fidelity
- [ ] Step 4: Design task ladder (recognition → comprehension → generation → formalization)
- [ ] Step 5: Define per-task evaluation metrics, including execution-based ones
- [ ] Step 6: Select diverse models to benchmark (closed + open source, varying scale)
- [ ] Step 7: Run evaluation, report metrics per task
- [ ] Step 8: Analyze failure modes with case studies
- [ ] Step 9: Test whether scaling/finetuning closes the gap

Step 1: Find or construct the executable domain. Look for domains where output can be mechanically verified without physical execution: DSLs, code, structured markup, CAD scripts, music notation, knitting/crochet patterns, circuit diagrams. Avoid domains where verification requires real-world execution (e.g., cooking, physical assembly) unless a symbolic proxy exists.

Step 2: Build paired dataset at scale.

• Source raw real-world artifacts (PDFs, images, web pages) paired with final visual outputs.
• Use an LLM-based extraction pipeline (e.g., GPT-4o-mini) to parse unstructured source into structured JSON with a unified schema (metadata, materials/components, step-by-step instructions).
• Target thousands of examples across many categories/difficulty levels for statistical power.

Step 3: Validate extraction fidelity. Sample a few hundred examples across all categories. Compute word-level overlap between parsed structured data and original raw source, focusing especially on procedural/instructional sections. Report both quantitative overlap (%) and manual spot-checks. This validates your data pipeline is trustworthy before benchmarking.

Step 4: Design a task ladder of increasing difficulty. Structure tasks so each isolates one cognitive ability:

• Task A (Recognition): multi-label classification of components/primitives from image (e.g., stitch types, parts). Real-world composites usually have MULTIPLE co-occurring labels — treat as multi-label, not single-label.
• Task B (Comprehension): vision-to-text grounding via forced-choice MCQ. Draw distractors from the same category as the correct answer to prevent superficial lexical shortcuts. Verify the answer distribution is balanced (roughly uniform across options) to rule out positional bias.
• Task C (Generation): free-form instruction generation from image, evaluated with text-similarity metrics as a baseline (necessary but insufficient).
• Task D (Formalization): translate into an executable representation. Evaluate at both a fine-grained step level (small, expert-annotated eval set) and a full-project level (large-scale, automated).

Step 5: Define metrics — always pair surface metrics with execution metrics.

• Surface: precision/recall/F1 (classification), accuracy (MCQ), BLEU/ROUGE/ChrF (generation).
• Execution: does it compile? Valid Pattern/Program Rate. Structural/perceptual similarity between executed output and ground truth (e.g., visual similarity via embedding distance like DINO similarity for rendered outputs).
• This dual-metric design is what reveals the "describing vs. doing" gap.

Step 6: Select a broad model panel. Include closed-source frontier models (GPT-4o, Gemini, Claude) and open-source models spanning a wide parameter range (e.g., 3B–72B) to test whether scale correlates with procedural competence.

Step 7-8: Run, report, and diagnose. Report metrics per task per model. Include at least one qualitative case study showing a model producing locally fluent, plausible-looking output (correct colors, correct local motifs) that fails globally (wrong overall structure) — this is often the most illustrative failure mode.

Step 9: Test whether scale/finetuning helps. Explicitly check: (a) do larger models perform better on execution-based metrics? (b) does supervised finetuning improve execution metrics, or only surface metrics? Report negative results directly — a common finding is that finetuning boosts BLEU but not valid-execution-rate, and that larger models sometimes hallucinate more novel/invalid symbols, hurting execution rate despite fluent output.