Applying 3D Reasoning Frameworks
Given a task, map it to the framework whose core mechanism matches:
| Task signal | Framework | Core mechanism |
|---|---|---|
| Need dense 3D segmentation mask from text prompt + point cloud | Reason3D | Aligns point features with text embeddings, hierarchical search over point cloud regions |
| Need spatial reasoning from limited 2D views without any 3D labels | 3DThinker | Latent 3D mental imagery embeddings inside a VLM's reasoning chain |
| Need an agent to actively manipulate/explore a 3D scene to answer a question | Think3D | Interactive 3D chain-of-thought + RL-learned exploration policy using manipulation tools |
| Need to infer occluded/unseen content or relations not visible in current frame | MindJourney | World-model-based imagined 3D exploration, iterative search over generated views |
| Need to ground implicit/indirect instructions to a specific 3D object location | ReGround3D | Chain-of-Grounding: alternates reasoning and grounding steps to localize objects |
Progress:
- Step 1: Identify the input modalities available (point cloud, single image, multi-view, video)
- Step 2: Identify the output required (segmentation mask, bounding box/location, free-text answer, exploration action)
- Step 3: Determine whether explicit 3D data exists or must be inferred/imagined
- Step 4: Match to framework(s) using the decision criteria below
- Step 5: Note training requirements (RL, latent embedding pretraining, paired 3D-text data) before recommending
- Step 6: If task spans multiple needs (e.g., grounding + segmentation), recommend a hybrid or pipeline of two frameworks
Decision criteria (in order of priority)
-
Is a real 3D point cloud available as input?
- Yes, and the goal is a dense mask → Reason3D
- Yes, and the goal is object localization from ambiguous/implicit instructions → ReGround3D
- No (only images/video) → go to 2
-
Is only 2D/limited-view visual input available, and 3D structure must be inferred internally?
- Goal is reasoning about spatial relations/answering questions → 3DThinker (no external 3D priors, lightweight latent embedding)
- Goal is inferring what's beyond the visible frame via generative simulation → MindJourney (uses an explicit world model to generate/search imagined views)
-
Does the task require the agent to take actions (move, zoom, rotate, query tools) to gather more 3D information?
- Yes → Think3D (interactive CoT with manipulation tools, RL-trained exploration)
-
Does the instruction require multi-step alternation between reasoning and locating?
- Yes → ReGround3D's Chain-of-Grounding pattern, even if borrowed as a design pattern within another system
Example 1: Input: "Segment the chair the person is most likely to sit on, given a room-scale point cloud and the instruction 'the seat closest to the window.'" Output: Recommend Reason3D. Rationale: real point cloud input + implicit spatial instruction + dense segmentation mask output; Reason3D's point-text feature alignment and hierarchical search directly targets this case.
Example 2: Input: "From a single photo of a kitchen, answer: 'Is the stove closer to the sink or the fridge?' No 3D scan available." Output: Recommend 3DThinker. Rationale: only limited 2D view, no labeled 3D data or point cloud; task needs internal spatial reasoning, which 3DThinker performs via latent mental 3D imagery rather than explicit 3D input.
Example 3: Input: "Given a single frame of a hallway, determine what room is likely behind the closed door on the left." Output: Recommend MindJourney. Rationale: requires inferring content beyond the visible frame; MindJourney's world-model-based imagined exploration is designed exactly for extrapolating unseen space.
Example 4: Input: "Build an agent that can rotate/zoom a rendered 3D object and decide, step by step, whether it's a mug or a vase." Output: Recommend Think3D. Rationale: requires active manipulation tools and an autonomous exploration policy, which is Think3D's defining feature (RL-trained interactive 3D CoT).
Example 5: Input: "User says 'grab the thing next to what I usually put my coffee on' — locate the referenced object in a scanned room." Output: Recommend ReGround3D. Rationale: implicit, indirect human instruction requiring joint reasoning before grounding; Chain-of-Grounding mechanism alternates reasoning/grounding to resolve such references.
- Distinguish "explicit 3D input available" vs. "3D must be imagined/inferred" first — this single question eliminates half the candidates immediately.
- When instructions are implicit/indirect (not naming the object directly), prefer frameworks with explicit reasoning-before-localization steps (ReGround3D) over one-shot grounding.
- For any framework requiring RL (Think3D) or world models (MindJourney), flag that these need more training infrastructure/compute than embedding-alignment approaches (Reason3D) or latent-only methods (3DThinker).
- When a pipeline needs both grounding and dense masks, consider chaining: use a ReGround3D-style reasoning-grounding loop to get a coarse location, then Reason3D-style segmentation to refine to a mask.
- Always check whether the deployment has real sensor/point-cloud data; if not, immediately narrow to 3DThinker or MindJourney rather than forcing a point-cloud method with synthetic reconstructions.
- Don't apply Reason3D-style methods when no point cloud is available — they depend on point-level features, not synthesized geometry.
- Don't use 3DThinker when a dense segmentation mask is required — it produces latent reasoning embeddings, not pixel/point-level outputs.
- Don't conflate MindJourney's generative world-model exploration with Think3D's tool-based manipulation — the former imagines unseen space, the latter acts on/explores an available 3D representation via tools.
- Don't select ReGround3D purely for "3D grounding" tasks with explicit, unambiguous instructions — its added value is specifically handling implicit instructions via alternating reasoning/grounding; simpler grounding tasks don't need this overhead.
- Don't assume any of these frameworks are interchangeable "3D LLM" black boxes — each targets a distinct combination of input modality, reasoning style, and output type.