Producing Roblox 3D Animations
Given a script/narration + reference images, produce a video by running this loop:
1. Analyze inputs (narration audio, images, video refs) → extract style, timing, events
2. Build storyboard: scene list with duration, camera, characters, environment, lighting
3. Resolve assets: search Kenney/KayKit CC0 libraries → fallback to procedural generation
4. Implement scenes: R3F + @remotion/three components, deterministic frame-based animation
5. Sync audio: map narration segments/pauses to scene cut points and character actions
6. Render preview frames → visually inspect → diff against reference → fix defects
7. Chunk-render final video with checkpoints → FFmpeg mux audio → validate output
Never skip step 6. Code executing without errors ≠ scene complete.
Progress:
- [ ] Phase 1: Input Analysis — parse script, narration audio, images, video refs
- [ ] Phase 2: Production Planning — storyboard, scene breakdown, resource estimate
- [ ] Phase 3: Asset Preparation — search/verify/license CC0 assets, prep R6 character
- [ ] Phase 4: Scene Implementation — R3F scenes, deterministic animation code
- [ ] Phase 5: Audio Synchronization — map narration timeline to scene events
- [ ] Phase 6: Preview & QA — render previews, visual inspection, defect correction
- [ ] Phase 7: Final Rendering — chunked render, checkpoints, FFmpeg assembly, validation
Phase 1 — Input Analysis
- Transcribe/measure narration: total duration, segment boundaries, pauses, emphasized words.
- For each reference image: extract style (low-poly/Roblox-blocky/realistic), palette, lighting mood, camera angle, composition.
- For reference video: sample frames at scene-change points, log timestamp + description of camera move, action, and framing.
- List missing information (e.g., "no character reference for Villager A") and ask only essential questions before proceeding.
Phase 2 — Production Planning
Produce a scene table. Each row is a concrete spec, not prose:
| Scene | Narration slice | Duration (frames@30fps) | Environment | Characters/Actions | Camera | Lighting/FX |
Rule: every scene must advance the narration. Cut static/filler shots.
Phase 3 — Asset Preparation
- Search order: Kenney → KayKit → other CC0 sources → procedural generation as last resort.
- For every asset used, record in an asset registry (JSON):
{ name, source, url, license, format, triangleCount, modifications }. - Verify license permits the intended use (commercial/redistribution if applicable).
- Convert/optimize to GLB, decimate geometry, bake/simplify materials for CPU (SwiftShader) rendering — target low poly counts and texture sizes appropriate for 4-core/no-GPU rendering.
- Use the existing rigged Roblox R6 GLB as the canonical character; never substitute a different rig without approval.
Phase 4 — Scene Implementation
- All animation driven by absolute frame number (
framefrom Remotion'suseCurrentFrame), neverDate.now(), never accumulated deltas, neverMath.random()without a fixed seed derived from frame/scene index. - Structure: one React Three Fiber component per scene, composed inside Remotion
<Sequence>blocks. - Character motion: procedural skeletal animation (bone rotations as pure functions of frame) layered with any baked clips from the GLB.
- Camera: keyframed position/lookAt as a function of frame, eased with deterministic easing functions (no physics/randomness-based cameras).
- Keep each scene module self-contained (own folder: scene component, asset list, animation functions) so scenes can be rendered/tested independently.
Phase 5 — Audio Synchronization
- Treat narration audio as immutable ground truth — never time-stretch or pitch-shift it.
- Build a timeline map:
{ narrationSegmentId, startFrame, endFrame, sceneEvents: [...] }. - Align character gestures, camera cuts, lighting changes, and SFX triggers to segment boundaries and emphasis points, not arbitrary fixed intervals.
Phase 6 — Preview & QA
For every scene, before moving on:
- Render a handful of representative frames (start, mid, end + any complex-action frame).
- Inspect for: broken/missing geometry, missing materials/textures, clipping, incorrect character proportions, skeletal glitches, camera framing errors, lighting mismatches vs. reference.
- Compare side-by-side against the reference image/video and the scene spec from Phase 2.
- If defective: diagnose root cause (asset issue, transform math, missing material assignment, wrong frame calc) and fix the source, not the symptom.
- Re-render and re-verify before marking the scene done.
- If a defect can't be resolved automatically, report it explicitly — do not ship it silently.
Phase 7 — Final Rendering
- Render in fixed-size chunks (e.g., 150–300 frame segments) with overlap/guard frames at boundaries to avoid seams.
- Write a checkpoint file per completed chunk; on resume, scan for missing/duplicate frames before re-rendering anything.
- Assemble with FFmpeg: concatenate video chunks, mux original narration audio + SFX, verify output resolution (1280×720), FPS (30), total duration matches narration length, and audio/video sync drift is negligible.
- Produce a production report: scene list, assets used (with licenses), known issues, render time, validation results.
Example 1:
Input: Narration about the Great Fire of London (1666) + historical street photos + one Roblox-cinematic reference video.
Output: Storyboard of 8 scenes (market street establishing shot → fire ignition → spreading flames/smoke with fleeing R6 characters → river evacuation → aftermath), asset registry listing Kenney medieval building pack + procedural fire particle system, scene components in src/scenes/fire-of-london/, QA log showing 3 fixed defects (clipping building geometry, mismatched smoke color, camera clipping through wall), final output.mp4 at 1280×720/30fps matching narration duration of 118s.
Example 2: Input: Script for a two-character dialogue scene with no visual references provided. Output: AI asks one essential question ("Any preferred environment/setting, or should I default to a neutral Roblox plaza?"), then proceeds autonomously through all 7 phases without further prompts, delivering the rendered scene plus report.
- Default to searching for existing CC0 assets before modeling anything new.
- Keep scene modules independent so any single scene can be re-rendered/tested in isolation.
- Log every technical decision (asset substitutions, performance compromises, style interpretations) in the production report.
- Treat the reference material as the spec — when in doubt, match the reference over inventing new style choices.
- Favor fewer, purposeful camera moves over constant motion; stillness is fine if narration doesn't demand action.
- Keep polycount/texture budgets conservative given CPU-only rendering constraints.
- Treating "code runs without error" as "scene complete" — always visually verify.
- Using
Date.now(), random seeds, or frame-delta accumulation — breaks determinism and chunked/resumable rendering. - Time-stretching or speeding up narration to fit scene pacing — never alter the source audio.
- Modeling assets from scratch when a suitable CC0 asset exists.
- Re-rendering completed chunks from scratch after an interruption instead of resuming via checkpoints.
- Silently accepting visual defects (clipping, broken rigs, missing materials) instead of reporting unresolved issues.
- Changing core architecture (swapping Three.js, dropping Remotion, reducing resolution/FPS) without explicit approval.