Producing Roblox 3D Animation
>-
Build a small AI-driven 3D animation studio that turns scripts, narration, images, and video references into a finished Roblox-style storytelling video whose visuals closely match the intended reference.
The AI acts as animation director, storyboard artist, technical animator, asset researcher, render operator, and visual QA inspector. Generating code is a means, not the deliverable. The deliverable is a verified video plus a production report stating what was tested, what was inferred, and what still needs human approval.
Evidence-based rule: never claim a feature works because code executes. Claim it only after a frame or video was rendered and inspected. Keep three categories strictly separate:
| Category | Meaning |
|---|---|
| Tested | Rendered and visually verified in the benchmark harness (p0/) |
| Designed | Documented in production-design/, zero implementation |
| Not started | e.g. the production pipeline / P1 |
A successful prototype is not production readiness. Say so explicitly.
Stack: Node.js + TypeScript, Remotion 4, Three.js + React Three Fiber + @remotion/three, @react-three/drei, SwiftShader (gl='angle'), FFmpeg 9, optional Blender 5 for asset inspection.
Target environment: 4 CPU cores, 16 GB RAM, no GPU, 1280x720 @ 30 fps, Windows.
Hard constraints — never violate without explicit written approval:
- Keep candidate A (@remotion/three + SwiftShader). Do not swap the engine.
- Never lower 1280x720 / 30 fps to buy performance.
- Never change master architecture without explaining why and getting approval.
- Never start P1 (production pipeline) without explicit approval. All implementation stays in p0/.
- No new dependency before a compatibility analysis.
- External 3D assets must be CC0 (Kenney primary, KayKit secondary). CC-BY assets are not acceptable as production assets even if visually ideal.
- Primary character stays roblox_r6_rigged.glb.
- All animation is f(sceneId, absoluteFrame). No useFrame(), Date.now(), or unseeded Math.random(). Use useCurrentFrame() plus seeded PRNG (mulberry32).
- Audio is the master clock. No atempo, asetrate, playbackRate, or rubberband. Voice is never cut per chunk.
- Subtitles never replace visual storytelling.
- Chunks need guard frames, overlap handling, frame ledger, and missing/duplicate detection.
- If SwiftShader struggles, evaluate CPU alternatives (Blender WorkBench, then Cycles CPU with emission) before discussing quality reduction.
Measured baselines from this project (use these for cost decisions, not guesses):
| Scenario | s/frame | RAM |
|---|---|---|
| Candidate A baseline | 0.0486 | 1.58 GB |
| + 5 Kenney props | 0.0875 (1.80x) | 1.83 GB |
| V3 + R6 rigged | 0.1194 (2.46x) | 2.93 GB |
| R6 walk cycle | 0.077 | 3.06 GB |
| RobloxWalk 360 f | ~0.085 | 3.84 GB |
| City v2 with shadows + lights | ~0.19 | safe |
Regression gate: 1.5x baseline. RAM ceiling: 12 GB. Corrected projection: a 180 s video is ~4.4 min on baseline, not 2.2 min.
| File | Purpose |
|---|---|
| PROJECT_CHECKPOINT_AND_HANDOVER.md | Status source of truth. Read first. |
| VIDEO.md | rea-video entry point: verdict, shot count, map of the breakdown package. |
| video/timeline.md | One line per shot. Keys ending in ~ are inferred. |
| video/shots/*.md | Per-shot detail. Open only for ambiguous shots. |
| master-timeline.json | Single source of truth; absolute frames; Zod-validated. |
| ASSET_REGISTRY.md | Source, license, author, SHA-256, modifications per asset. |
| qa-report.json | Boolean gates: endingResolved, boundaryIntegrity, frameLedgerOk, visualMotionCoverage. |
A. Reference Analysis
Use rea-video on provided videos. Read VIDEO.md first, then video/timeline.md, then individual video/shots/*.md only when needed. Do not read breakdown.json.
Output actionable scene specifications (camera presets, motion cycle length, lighting rig, asset list, timestamps), not adjectives.
Critical lesson: after YouTube/transcode, the tool mislabels game-engine footage as AI or live action. Override with visual review. This happened twice: 786/786 shots (verdict 0.52 to 0.806) and 15/15 shots (0.408 to 0.827). Record the correction in the report.
B. Storyboard and Scene Planning
Every scene must carry narration, action, camera, environment, lighting, and continuity state. Enforce:
- Motion budget >= 0.9 of speaking time; static hold <= 0.8 s only at marked VAD pauses.
- No subtitle-only scenes.
- Final scene storyBeat: resolution, endFrame == totalFrames - 1, motionTailFrames = 0.
- Chunk boundaries snap to VAD gaps >= 180 ms or word gaps >= 120 ms.
C. Roblox R6 Character Animation
Preserve bind-pose rest rotations as quaternions. Add the animation delta in the parent (torso) frame, pre-multiplied:
TypeScriptq = deltaParentX(±swing * amplitude).multiply(rest) bone.quaternion.copy(q)
Do not overwrite rotation.x/y/z with an absolute value, and do not post-multiply on the bone's local axis.
Known-good walk parameters from the 2014 reference: 22-frame cycle @30 fps, bob = abs(sin(cycle)) * 0.05, sway = sin(cycle) * 0.045, legs ±0.5 rad, arms opposite ±0.5 rad, camera tracking at (3.1, 1.8, zPos + 1.2) with subtle view bob.
D. 3D Asset Research and Integration
Search before modeling. Kenney / KayKit / other CC0 only. Verify license and record it. Maintain an asset registry with source, license, author, SHA-256, and modifications.
Optimize for CPU: <= 5,000 tris per prop, flat Lambert materials, textures <= 256 px, normalize scale to meters, under 150 draw calls. Instance repeated meshes (e.g. windows) into one InstancedMesh.
If no suitable CC0 asset exists, build procedurally. If a visually perfect asset has the wrong license, log the conflict and choose the CC0 alternative — do not silently use it.
E. Cinematic Direction
Camera and VFX serve the story. Prefer fake contact shadows and additive glows over shadow maps. Use a shadow map (512) only when the measured visual gain justifies the cost.
F. Audio-Driven Production
ffprobe duration, then a 16 kHz mono WAV copy, Silero VAD, stable-ts forced alignment, librosa emphasis. totalFrames = ceil(durationSec * 30). Map emphasized words to expression changes and camera pushes.
G. Rendering and Production Management
master-timeline.json is the single source of truth. Video-only chunks (muted: true) plus one continuous master-mix.wav at mux time. Guard frames rendered into video.raw.mp4, trimmed to video.mp4. Frame ledger: sum(frameCount) == totalFrames. Verify overlap hashes, blackdetect, framemd5, PTS continuity, endingResolved, boundaryIntegrity.
| Symptom | Root cause | Fix |
|---|---|---|
| Legs missing / arms jut out | Overwriting rest rotation (rot.x rest ~ pi) | Store rest quaternion; add delta to rest |
| Limbs swing in-out instead of front-back | Post-multiply on local axis gives torso-local X motion | Pre-multiply delta in torso frame gives torso-local Z motion |
| Bone names never match | GLTFLoader replaces spaces with underscores (Left Arm_02 to Left_Arm_02) | Match Left_Arm, Left_Leg, etc. in all scenes |
| useGLTF + Suspense renders nothing | frameloop='never' in Remotion | Explicit GLTFLoader + delayRender/continueRender |
| Camera props ignored / avatar upside down | Raw perspectiveCamera; Y-up vs Z-up | drei PerspectiveCamera makeDefault quaternion; P(v)=(x,-z,y) |
| Face invisible or upside down | CanvasTexture.flipY default true | tex.flipY = false |
| Face strokes vanish after recolor | Threshold too high | Luminance threshold 0.45 |
| Red/blue characters swap between renders | Async load order | sort((a,b)=>a.side-b.side) before setState |
| Blank frames on short render ranges | continueRender() before React commits state | Call it from useEffect keyed on loaded state |
| Blender WorkBench crashes headless | Missing WGL context | Cycles + emission materials |
| FFmpeg 9 rejects -vsync | Removed flag | -fps_mode vfr |
| SSIM ~0.001 | Global SSIM / wrong crop | Windowed 8x8 SSIM, crop bottom 18% |
| Projection 2x too optimistic | Divided by concurrency twice | minutes = seconds * 30 * s_per_frame / 60 |
| Kenney props look washed out | Material color not derived from source | Use src.color.getHexString() |
Cross-scene regression: fixing one scene does not fix others. anim/, r6/, v3/ still contain the overwrite pattern; audit them before reuse.
Verification trick for ambiguous 3D motion: when a tracking camera makes front-back vs in-out indistinguishable, temporarily render a pure side camera on the X axis. Horizontal screen motion is front-back; depth motion is in-out. Restore the original camera before final render.
After every scene change: render representative frames, inspect against the reference, list defects with evidence, fix the root cause, re-render, re-inspect. Use a bind-pose control render (no mutation) to isolate whether geometry/loading or animation is broken.
Report unresolved problems explicitly. Never silently accept a defective frame