Building Anime Story Video Editor
Building Anime Story Editor
Scaffold runs as a sequence of module builds, not a single generated blob. Build in this order and verify each module runs standalone before wiring the pipeline:
app/config.py -> paths, model sizes, CPU thread limits
app/audio/transcribe.py -> faster-whisper wrapper, returns sentence-level segments
app/vision/scenes.py -> PySceneDetect wrapper, returns per-clip scene list + thumbnails
app/vision/describe.py -> OpenCLIP embeddings for frames + text (voiceover sentences)
app/matching/match.py -> cosine-similarity matching, anti-repeat logic, timeline builder
app/editing/timeline.py -> neutral JSON timeline schema + validation
app/editing/render.py -> FFmpeg cut/concat/subtitle/mix/export
app/exporters/srt.py -> SRT writer
app/exporters/kdenlive.py -> optional .kdenlive XML writer
app/main.py -> CLI orchestrator with progress + resume
Run with:
python app/main.py
Progress checklist to track while building:
- [ ] Verify environment: python, ffmpeg -version, pip packages installable on target CPU
- [ ] Create folder structure (workspace/input/{voiceover,clips,music}, analysis, previews, output)
- [ ] Implement config.py (CPU thread caps, model names, default aspect ratio)
- [ ] Implement transcription module + test on a short sample WAV/MP3
- [ ] Implement scene detection + frame extraction + test on one sample clip
- [ ] Implement CLIP embedding for frames and sentences + test similarity on dummy data
- [ ] Implement matching logic with anti-repeat + continuity rules
- [ ] Define neutral timeline JSON schema, write validator
- [ ] Implement FFmpeg render pipeline (cut, concat, subtitle burn/soft, audio mix)
- [ ] Implement SRT exporter
- [ ] Implement optional Kdenlive XML exporter
- [ ] Implement main.py CLI: input detection, missing-file messages, stage-by-stage resume
- [ ] Add logging (file + console) at every stage
- [ ] Write unit tests for transcribe, scenes, matching, timeline validation
- [ ] Create requirements.txt, install.bat, run.bat, README.md
- [ ] Run end-to-end on sample media if available; report actual results, not assumptions
Neutral Timeline JSON Schema (source of truth)
JSON{ "version": 1, "aspect_ratio": "16:9", "voiceover_file": "workspace/input/voiceover/vo.mp3", "music_file": "workspace/input/music/bgm.mp3", "sentences": [ { "id": 0, "text": "The village burned that night.", "start": 2.14, "end": 4.80, "emotion": "sad", "keywords": ["village", "burned", "night"], "selected_scene": { "clip_file": "workspace/input/clips/ep1.mp4", "scene_start": 112.3, "scene_end": 114.9, "confidence": 0.78, "needs_review": false } } ], "subtitle_style": {"font": "Arial", "size": 42, "highlight_color": "#FFD700"} }
Every render step reads only from this JSON. Regenerating the video from an edited JSON must never require re-running transcription or vision analysis — cache those in workspace/analysis/.
CPU-Friendly Model Defaults (i5-4570, 16GB RAM, no GPU)
- faster-whisper:
smallorbasemodel,compute_type="int8",cpu_threads=4 - OpenCLIP:
ViT-B-32withlaion2b_s34b_b79kweights (small, CPU-tolerable); batch frame embedding in groups of 8-16 - PySceneDetect:
ContentDetectorwith threshold ~27, downscale factor 2 for speed - Frame extraction: 1 representative frame per detected scene, not every frame
- FFmpeg:
-preset medium,-crf 20,libx264,yuv420p, audioaac 192k
Example 1:
Input: User places vo.mp3 (30s narration about a hero's journey) and 6 anime clips in input folders, runs python app/main.py.
Output: Console shows stages: "Transcribing voiceover... 8 sentences found", "Analyzing clip 1/6... 14 scenes detected", ..., "Matching scenes to narration...", "Rendering final video...", then workspace/output/final_video.mp4, workspace/output/subtitles.srt, workspace/output/timeline.json appear.
Example 2:
Input: User forgets to add any clips.
Output: CLI prints exactly: ERROR: No video clips found in workspace/input/clips/. Add 5-10 .mp4/.mkv anime clips and run again. and exits cleanly without crashing or creating partial output.
- Treat voiceover sentence timestamps as immutable; everything else adapts to them.
- Cache every expensive stage (transcripts, scene lists, embeddings) to
workspace/analysis/<hash>/keyed by file hash+mtime, so reruns skip unchanged inputs. - Never mutate files under
workspace/input/; copy toworkspace/previews/for any trimmed preview. - Before any destructive operation on
workspace/analysisor timeline JSON, copy previous version to*.bak. - Log every matching decision (sentence id, chosen clip, confidence, reason) so low-confidence matches are visible in a
review_needed.jsonthe user can skim. - Keep anti-repeat logic simple and explainable: track last N used (clip, scene) pairs, penalize reuse unless no better candidate exists above a similarity floor.
- Use subprocess calls to real
ffmpeg/ffprobebinaries; verify their presence and print install instructions if missing instead of failing with a stack trace. - Write small pure functions (duration calc, similarity scoring, SRT timestamp formatting) and cover them with unit tests using tiny synthetic fixtures, not large media.
- Default aspect ratio to 16:9 at 1080p if not specified; make it a config constant, not hardcoded in multiple places.
- Do not burn subtitles permanently without also saving a standalone
.srt— users need the editable version. - Do not re-run Whisper or CLIP on unchanged files; always check the analysis cache first, this is the difference between usable and unusable on this hardware.
- Do not select the same scene for consecutive sentences unless similarity scores clearly justify it — this breaks storytelling flow.
- Do not add transitions/zooms by default; only apply them when a config flag or clear pacing rule triggers it.
- Do not silently continue when
ffmpeg/ffprobearen't on PATH — fail fast with a clear message and installation steps for Windows. - Do not fabricate scene descriptions when CLIP confidence is low — mark
needs_review: trueand pick the best available candidate instead of inventing content. - Do not load the full video into memory for scene detection; stream/decode incrementally via OpenCV/PySceneDetect.
- Do not skip writing tests because media files are large — test logic functions with small synthetic inputs (fake timestamps, fake embeddings) instead.