Benchmarking VLM Crochet Pattern Understanding

Bash
# 1. Run a single model on a single task
python benchmark_task/task_a_gemini.py --input data/selected_crochet_patterns_3.json --output results/task_a_gemini.jsonl

# 2. Evaluate the task's outputs
python benchmark_task/eval_task_a.py --predictions results/task_a_gemini.jsonl --output results/task_a_gemini_scored.csv

# 3. Compare rendered images against ground-truth photos across all models
python batch_compare_images.py --patterns data/selected_crochet_patterns_3.json --img_dirs render_outputs/ --output csv_results/

Progress:

• Step 1: Define the task family (Task A/B/C/D-project/D-step) and its input/output schema
• Step 2: For each task, write one script per model, sharing a common interface (utils.py)
• Step 3: Write one eval_task_X.py per task that scores all models' outputs against ground truth
• Step 4: For visual comparison tasks, batch-discover generated images and match them to dataset entries by ID/filename pattern
• Step 5: Load all similarity models once (LPIPS, DINO, CLIP, SSIM) and reuse across the batch
• Step 6: Write per-model CSV results with a timestamp column; never overwrite prior runs
• Step 7: Aggregate CSVs into a summary comparison table across models/metrics

Task family design pattern

• Task A: single-image → attribute/description generation (e.g., identify pattern from photo)
• Task B: text pattern → property extraction / classification
• Task C: multi-step reasoning or generation graded at output level
• Task D-project: full-project-level evaluation (holistic, multi-image)
• Task D-step: step-level evaluation (per-instruction-step grading)

Each task gets a separate script per model (task_a_claude.py, task_a_gemini.py, task_a_qwen_72b.py, etc.) rather than one script with a --model flag. This keeps model-specific SDK quirks (auth, rate limits, prompt formatting, multimodal input encoding) isolated and lets any single model's script fail without breaking the others. Share only generic helpers (I/O, prompt templates, retries) via utils.py.

Model script skeleton (replicate per model)

Python
# task_a_<model>.py
import json, time, argparse
from utils import load_dataset, save_jsonl, retry_with_backoff

def call_model(image_path, prompt, client):
    # model-specific API call, wrapped in retry_with_backoff
    ...

def main():
    args = parse_args()
    dataset = load_dataset(args.input)
    results = []
    for item in dataset:
        try:
            response = call_model(item["image_path"], build_prompt(item), client)
            results.append({"id": item["id"], "prediction": response})
        except Exception as e:
            results.append({"id": item["id"], "error": str(e)})
        time.sleep(args.rate_limit_delay)
    save_jsonl(results, args.output)

Image similarity pipeline design

Load every model once at startup (load_models(device)), never per-image-pair — model loading (LPIPS/DINO/CLIP) is expensive. Compute multiple complementary metrics per pair since they capture different notions of similarity:

• LPIPS: perceptual similarity (low-level texture/structure), lower distance = more similar
• DINO: self-supervised semantic feature similarity (cosine similarity of embeddings), good for shape/object-level match
• CLIP: semantic/text-aligned similarity, useful for coarse "is this the same object type"
• SSIM: structural similarity, sensitive to exact pixel alignment — use only for near-identical crops
• Wasserstein distance: color/histogram distribution comparison

Auto-discover rendering directories by convention (*_img/) and match files to dataset entries via ID embedded in filename — don't hardcode paths per pattern.