Building Anime Story Editor
Building Anime Story Editor
Builds a working, offline, free-tool video editing pipeline: voiceover + anime clips in → analyzed, matched, rendered MP4 out. No CapCut, no GPU requirement, no paid APIs.
If starting fresh, create the project in this order and verify each piece runs before moving on:
Bash# 1. Scaffold folders mkdir -p app/{audio,vision,matching,editing,exporters} workspace/input/{voiceover,clips,music} workspace/analysis workspace/previews workspace/output tests .opencode/skills/anime-story-editor # 2. Install deps (CPU-only, lightweight) pip install faster-whisper opencv-python scenedetect[opencv] open_clip_torch torch --index-url https://download.pytorch.org/whl/cpu numpy pillow ffmpeg-python srt # 3. Verify FFmpeg ffmpeg -version # 4. Run python app/main.py
Never claim a step works without actually running it on sample media first (generate a 5s synthetic clip + tiny WAV with ffmpeg if no sample media exists).
Progress:
- Inspect environment (Python version, ffmpeg presence, CPU-only torch availability, disk space)
- Scaffold full folder structure exactly as specified
- Implement
app/config.py(paths, model sizes, thresholds — all CPU/i5-4570 tuned) - Implement
app/audio/transcribe.py(faster-whisper,small/baseint8, sentence+word timestamps, emotion/keyword heuristics) - Implement
app/vision/scene_detect.py(PySceneDetect ContentDetector, extract representative frame per scene via OpenCV) - Implement
app/vision/frame_describe.py(OpenCLIP embeddings per frame; optional lightweight tag vocabulary for human-readable scene labels — never invent descriptions beyond what embeddings/tags support) - Implement
app/matching/match_engine.py(cosine similarity between sentence text embeddings and frame embeddings via OpenCLIP text/image tower; anti-repetition penalty; continuity bonus; uncertainty flagging) - Implement
app/editing/timeline_builder.py(produce neutral timeline JSON: schema below) - Implement
app/editing/subtitles.py(SRT writer, word-highlight ASS optional) - Implement
app/exporters/render_ffmpeg.py(concat/trim via ffmpeg filter_complex, audio mix voiceover+music with sidechain ducking, burn or soft subtitles, 1080p default) - Implement
app/exporters/kdenlive_export.py(generate.kdenliveXML referencing original clips + cut points) - Implement
app/main.py(CLI orchestrator: detect inputs, run pipeline stage by stage, resume fromworkspace/analysis/*.jsonif present, clear progress logs) - Write
tests/covering transcription parsing, scene detection on sample clip, matching logic, timeline JSON schema validation, ffmpeg command building - Write
install.bat/run.batfor Windows - Write
README.mdwith setup + usage - Run full pipeline end-to-end on minimal synthetic sample (silence+tone WAV, 2 color-bar clips) to prove it executes without crashing
- Report final folder tree, install steps, run command, input locations, output location
Neutral timeline JSON (source of truth, workspace/analysis/timeline.json):
JSON{ "aspect_ratio": "16:9", "fps": 30, "segments": [ { "index": 0, "voiceover_text": "The village was burning.", "vo_start": 2.14, "vo_end": 4.80, "emotion": "tense", "source_clip": "clips/ep3.mp4", "source_start": 112.3, "source_end": 114.9, "match_confidence": 0.78, "flagged_for_review": false, "transition": "cut" } ], "subtitles_path": "output/story.srt", "music_path": "input/music/theme.mp3" }
All rendering and exporting reads only this file — never recompute matches during render.
Resumability: persist after each stage: analysis/transcript.json, analysis/scenes.json, analysis/embeddings.npz, analysis/timeline.json. On rerun, main.py skips stages whose output file already exists and is valid JSON/NPZ, unless --force is passed.
Backups: before writing/overwriting any file in workspace/analysis or workspace/output, copy existing one to *.bak.
Model choices (i5-4570, 16GB RAM, no GPU):
- Whisper:
faster-whispersmallmodel,device="cpu",compute_type="int8". Fall back tobaseifsmallis too slow (log timing, let user choose via config). - OpenCLIP:
ViT-B-32withlaion2b_s34b_b79kweights — smallest practical model with good CPU speed. - Never load multiple heavy models concurrently; process sequentially and free memory (
del model; gc.collect()).
Matching algorithm:
- Embed each voiceover sentence (text tower).
- Embed each candidate scene's representative frame (image tower).
- Score = cosine_similarity − repetition_penalty(recent_use) + continuity_bonus(prev_clip_same_source).
- If top score < threshold (e.g. 0.20), mark
flagged_for_review: trueand pick best available rather than failing. - Enforce minimum scene duration (e.g. 1.2s) and don't reuse a scene within N segments unless no alternative exists.
FFmpeg rendering:
- Build filter_complex: trim each source segment → scale/pad to target resolution → concat.
- Mix audio: voiceover at 0dB, music ducked via
sidechaincompressor fixed -18dB under voiceover, SFX only if explicitly provided. - Burn subtitles with
subtitles=filter or mux as soft subs (.srtalongside for editability) — default to soft subs + provide a burned-in optional flag. - Always render to a temp file then move to
workspace/output/to avoid partial corrupt files.
Example 1:
Input: workspace/input/voiceover/story.wav (45s), 6 clips in clips/, no music.
Output: workspace/output/story_final.mp4 (1080p, 16:9), workspace/output/story.srt, workspace/analysis/timeline.json listing 14 matched segments, 2 flagged for review with a printed CLI warning listing their indices and timestamps.
Example 2:
Input: Rerun after adding music to input/music/.
Output: Pipeline detects existing transcript.json/scenes.json/timeline.json, skips transcription and scene detection, only re-runs audio mixing + render with music added — logged as "Skipping transcription (cached) / Skipping scene detection (cached) / Re-rendering with music".
- Log every stage start/end with elapsed time and item counts (e.g. "Detected 143 scenes across 6 clips in 38s").
- Validate ffmpeg and ffprobe are on PATH before doing any work; fail fast with install instructions if missing.
- Check each input file opens correctly (OpenCV
VideoCapture.isOpened(), wave header valid) before processing; report exact filename and problem. - Keep CLI output readable: stage headers, progress bars for per-file loops (
tqdm), final summary table of segments + confidence. - Write small, pure functions per module so
tests/can import and assert directly without running the whole pipeline. - Default to safe, boring edits (hard cuts, no zooms) — only add a transition when a config flag enables it and it's not back-to-back.
- Don't re-decode full video files repeatedly — extract frames once during scene detection and cache thumbnails in
workspace/analysis/frames/. - Don't mutate anything under
workspace/input/— treat as strictly read-only. - Don't let a single corrupt clip crash the whole run — catch, log, skip that clip, continue with the rest.
- Don't hardcode absolute Windows paths — use
pathlib.Pathrelative to project root everywhere. - Don't silently guess scene content beyond what CLIP similarity/tags support — if uncertain, flag it, don't fabricate a description.
- Don't load the Whisper model and CLIP model into memory at the same time on this hardware — sequence the stages.