AI Skill Report Card

Building Anime Story Video Editor

A-82·Nov 12, 2026·Source: Web

Building Anime Story Editor

13 / 15

Scaffold runs as a sequence of module builds, not a single generated blob. Build in this order and verify each module runs standalone before wiring the pipeline:

app/config.py          -> paths, model sizes, CPU thread limits
app/audio/transcribe.py -> faster-whisper wrapper, returns sentence-level segments
app/vision/scenes.py    -> PySceneDetect wrapper, returns per-clip scene list + thumbnails
app/vision/describe.py  -> OpenCLIP embeddings for frames + text (voiceover sentences)
app/matching/match.py   -> cosine-similarity matching, anti-repeat logic, timeline builder
app/editing/timeline.py -> neutral JSON timeline schema + validation
app/editing/render.py   -> FFmpeg cut/concat/subtitle/mix/export
app/exporters/srt.py    -> SRT writer
app/exporters/kdenlive.py -> optional .kdenlive XML writer
app/main.py             -> CLI orchestrator with progress + resume

Run with:

python app/main.py
Recommendation▾
Add a second, more varied example showing a partial-failure or resume scenario (e.g., rerunning after editing timeline.json) to demonstrate caching behavior in practice.
14 / 15

Progress checklist to track while building:

- [ ] Verify environment: python, ffmpeg -version, pip packages installable on target CPU
- [ ] Create folder structure (workspace/input/{voiceover,clips,music}, analysis, previews, output)
- [ ] Implement config.py (CPU thread caps, model names, default aspect ratio)
- [ ] Implement transcription module + test on a short sample WAV/MP3
- [ ] Implement scene detection + frame extraction + test on one sample clip
- [ ] Implement CLIP embedding for frames and sentences + test similarity on dummy data
- [ ] Implement matching logic with anti-repeat + continuity rules
- [ ] Define neutral timeline JSON schema, write validator
- [ ] Implement FFmpeg render pipeline (cut, concat, subtitle burn/soft, audio mix)
- [ ] Implement SRT exporter
- [ ] Implement optional Kdenlive XML exporter
- [ ] Implement main.py CLI: input detection, missing-file messages, stage-by-stage resume
- [ ] Add logging (file + console) at every stage
- [ ] Write unit tests for transcribe, scenes, matching, timeline validation
- [ ] Create requirements.txt, install.bat, run.bat, README.md
- [ ] Run end-to-end on sample media if available; report actual results, not assumptions

Neutral Timeline JSON Schema (source of truth)

JSON
{ "version": 1, "aspect_ratio": "16:9", "voiceover_file": "workspace/input/voiceover/vo.mp3", "music_file": "workspace/input/music/bgm.mp3", "sentences": [ { "id": 0, "text": "The village burned that night.", "start": 2.14, "end": 4.80, "emotion": "sad", "keywords": ["village", "burned", "night"], "selected_scene": { "clip_file": "workspace/input/clips/ep1.mp4", "scene_start": 112.3, "scene_end": 114.9, "confidence": 0.78, "needs_review": false } } ], "subtitle_style": {"font": "Arial", "size": 42, "highlight_color": "#FFD700"} }

Every render step reads only from this JSON. Regenerating the video from an edited JSON must never require re-running transcription or vision analysis — cache those in workspace/analysis/.

CPU-Friendly Model Defaults (i5-4570, 16GB RAM, no GPU)

  • faster-whisper: small or base model, compute_type="int8", cpu_threads=4
  • OpenCLIP: ViT-B-32 with laion2b_s34b_b79k weights (small, CPU-tolerable); batch frame embedding in groups of 8-16
  • PySceneDetect: ContentDetector with threshold ~27, downscale factor 2 for speed
  • Frame extraction: 1 representative frame per detected scene, not every frame
  • FFmpeg: -preset medium, -crf 20, libx264, yuv420p, audio aac 192k
Recommendation▾
The JSON schema example is strong, but consider showing a before/after snippet of a 'needs_review' entry being resolved to make the matching workflow more concrete.
14 / 20

Example 1: Input: User places vo.mp3 (30s narration about a hero's journey) and 6 anime clips in input folders, runs python app/main.py. Output: Console shows stages: "Transcribing voiceover... 8 sentences found", "Analyzing clip 1/6... 14 scenes detected", ..., "Matching scenes to narration...", "Rendering final video...", then workspace/output/final_video.mp4, workspace/output/subtitles.srt, workspace/output/timeline.json appear.

Example 2: Input: User forgets to add any clips. Output: CLI prints exactly: ERROR: No video clips found in workspace/input/clips/. Add 5-10 .mp4/.mkv anime clips and run again. and exits cleanly without crashing or creating partial output.

Recommendation▾
Quick Start lists file paths but no actual runnable code snippet (e.g., a minimal config.py or main.py stub) — adding a few lines of real code would strengthen immediate actionability.
  • Treat voiceover sentence timestamps as immutable; everything else adapts to them.
  • Cache every expensive stage (transcripts, scene lists, embeddings) to workspace/analysis/<hash>/ keyed by file hash+mtime, so reruns skip unchanged inputs.
  • Never mutate files under workspace/input/; copy to workspace/previews/ for any trimmed preview.
  • Before any destructive operation on workspace/analysis or timeline JSON, copy previous version to *.bak.
  • Log every matching decision (sentence id, chosen clip, confidence, reason) so low-confidence matches are visible in a review_needed.json the user can skim.
  • Keep anti-repeat logic simple and explainable: track last N used (clip, scene) pairs, penalize reuse unless no better candidate exists above a similarity floor.
  • Use subprocess calls to real ffmpeg/ffprobe binaries; verify their presence and print install instructions if missing instead of failing with a stack trace.
  • Write small pure functions (duration calc, similarity scoring, SRT timestamp formatting) and cover them with unit tests using tiny synthetic fixtures, not large media.
  • Default aspect ratio to 16:9 at 1080p if not specified; make it a config constant, not hardcoded in multiple places.
  • Do not burn subtitles permanently without also saving a standalone .srt — users need the editable version.
  • Do not re-run Whisper or CLIP on unchanged files; always check the analysis cache first, this is the difference between usable and unusable on this hardware.
  • Do not select the same scene for consecutive sentences unless similarity scores clearly justify it — this breaks storytelling flow.
  • Do not add transitions/zooms by default; only apply them when a config flag or clear pacing rule triggers it.
  • Do not silently continue when ffmpeg/ffprobe aren't on PATH — fail fast with a clear message and installation steps for Windows.
  • Do not fabricate scene descriptions when CLIP confidence is low — mark needs_review: true and pick the best available candidate instead of inventing content.
  • Do not load the full video into memory for scene detection; stream/decode incrementally via OpenCV/PySceneDetect.
  • Do not skip writing tests because media files are large — test logic functions with small synthetic inputs (fake timestamps, fake embeddings) instead.
0
Grade A-AI Skill Framework
Scorecard
Criteria Breakdown
Quick Start
13/15
Workflow
14/15
Examples
14/20
Completeness
18/20
Format
14/15
Conciseness
13/15