Extracting Video Keyframes

Bash
# Extract 1 frame every 2 seconds from input video
ffmpeg -i input.mp4 -vf "fps=1/2" -qscale:v 2 frames/frame_%04d.png

# Get video metadata first (duration, fps, resolution) for timestamp mapping
ffprobe -v error -select_streams v:0 \
  -show_entries stream=width,height,r_frame_rate,duration \
  -of default=noprint_wrappers=1 input.mp4

Then score and select best frames (see Workflow), and output a manifest like:

JSON
{
  "source": "input.mp4",
  "interval_seconds": 2,
  "frames": [
    { "timestamp": "00:00:02", "file": "frame_0001.png", "score": 8.7, "tags": ["sharp", "face-visible", "high-contrast"] },
    { "timestamp": "00:00:04", "file": "frame_0002.png", "score": 4.2, "tags": ["blurry", "motion-blur"] }
  ],
  "best_frames": ["frame_0001.png", "frame_0009.png", "frame_0014.png"]
}

Progress:

• Step 1: Probe video (duration, fps, resolution, orientation)
• Step 2: Extract frames at fixed interval (default 2s; ask if user wants a different rate)
• Step 3: Score each frame (sharpness, exposure, composition, face/subject presence, scene-change weight)
• Step 4: Rank frames, dedupe near-identical ones, select top N or top-per-scene
• Step 5: Output structured manifest (JSON) + annotated contact sheet
• Step 6: Hand off best frames/timestamps to Remotion/Hyperframes for reel assembly
• Step 7: Export in requested formats (9:16, 1:1, 16:9) for reel/multi-platform use

Step 1 — Probe: Always run ffprobe first. Note duration, fps, resolution, orientation — these determine frame count and aspect-ratio crop strategy.

Step 2 — Extract: Use fps=1/N filter (N = interval in seconds, default 2). Save as PNG (lossless) for scoring, convert to JPG for lightweight previews if needed.

Step 3 — Score: For each frame compute:

• Sharpness (Laplacian variance) — penalize motion blur
• Exposure histogram — penalize over/under-exposed
• Face/subject detection presence — boost if relevant
• Scene-change delta vs. neighboring frames — boost distinct moments, suppress near-duplicates

Step 4 — Select: Group frames into scenes (via scene-change deltas), pick the top-scoring frame per scene rather than just the global top N — this avoids clustering all picks in one visually busy segment.

Step 5 — Output: Produce a JSON manifest (timestamp, filename, score, tags) plus a contact-sheet image (grid thumbnail) for quick human review.

Step 6 — Handoff: Map selected timestamps to a Remotion composition (e.g., <Sequence from={frame.timestamp}>) or Hyperframes input list, preserving original timecodes for re-sync with audio/source footage.

Step 7 — Multi-format export: For reels, re-crop/re-compose per target aspect ratio (9:16 vertical, 1:1 square, 16:9 landscape) using the same best-frame selections as anchor points for cuts/transitions.