One model learns appearance, depth, and camera motion through conditional completion. We explain how joint training helps, and how image-based verification selects among geometric predictions at inference time.
A photo or a short video shows only one slice of a scene: visible surfaces from one viewpoint at one moment in time. To understand the scene, we have to infer what the camera did not capture. How far away are the surfaces? Where did the camera move? What would we see from another viewpoint, or a moment later? These questions connect appearance, geometry, and time. We refer to this as 4D scene understanding.
The field has historically split this job in two. Geometric predictors estimate depth and camera pose from images. Video generators synthesize what a scene might look like as it evolves. Both have made substantial progress, but they are usually trained to answer different questions. A predictor typically returns one geometric estimate; a video generator can produce many plausible videos without explicitly recovering the geometry behind them.
We ask whether these capabilities can be learned together. RGB describes appearance, depth describes visible 3D structure, and camera pose describes viewpoint. Depending on what we observe, any of these can be an input or an output.
Our work, UGG: Unified Generative Geometry Model, treats generation and geometric prediction as conditional completion of the same scene. The same model generates video along a camera trajectory, predicts depth, and recovers camera pose. Changing the task changes which parts of the scene we reveal. This also lets us sample several possible geometric explanations and check them against the images at inference time.
For each frame, we represent the scene with an RGB image, a depth map, and a camera-ray map. RGB and depth already live on a pixel grid. A camera is usually represented by a small collection of parameters: position, rotation, and intrinsics, which describe how viewing directions map to pixels. We instead represent the camera by the ray passing through each pixel.
A ray tells us where the camera is and which direction that pixel looks into the world. We use a Plücker representation: a direction $\mathbf d$ and a moment $\mathbf m=\mathbf o\times\mathbf d$, where $\mathbf o$ is the camera center. This turns camera geometry into an image-shaped modality, aligned with RGB and depth. Depth and camera positions use a shared scene scale, so they describe compatible geometry.
The model can now exchange information between modalities at corresponding positions. A depth boundary, an image edge, and a change in viewing direction are represented on aligned grids rather than through unrelated interfaces.
We initialize from the Wan2.1-T2V-1.3B video model
A diffusion model learns to recover data from noise. We build on diffusion forcing,
To predict geometry from a video, we reveal the RGB frames and denoise depth and camera rays. To generate video along a camera path, we reveal that path and one or two RGB frames, then denoise the missing images and depth. With only the first image revealed, the model completes the rest of the scene sequence.
| Observed context | Missing components to infer |
|---|---|
| First RGB frame | Remaining RGB, depth, and camera rays |
| Full RGB sequence | Depth and camera rays |
| First and last RGB frames + camera trajectory | Intermediate RGB and depth |
| First RGB frame + camera trajectory | Remaining RGB and depth |
| First RGB frame + depth sequence | Remaining RGB and camera rays |
These are different conditioning patterns for the same weights. During inference, the observed tokens stay fixed at every denoising step, while the missing tokens are progressively recovered. We also use history guidance
The training objective is flow matching. For a clean token $z_0$ and Gaussian noise $\epsilon$, we form
\[z_t=(1-t)z_0+t\epsilon.\]The network predicts the velocity $\epsilon-z_0$. Different tokens can have different values of $t$, allowing clean observations and noisy targets to coexist in one sequence.
This matters for the data as well. Most video datasets do not provide RGB, depth, camera pose, and text together. We train on eight real, synthetic, and robotic datasets with different subsets of annotations, computing a loss only where a target is available. RGB-only clips contribute visual supervision; clips with camera or depth annotations also supervise those modalities. Missing labels do not need to be invented to fit a common training format.
The training mixture comprises RealEstate10K, DL3DV-10K, SpatialVID, ScanNet++, AgiBot World, OmniWorld, Virtual KITTI 2, and PointOdyssey. DL3DV-10K and ScanNet++ use depth and camera annotations from the VIPE pipeline; this is training supervision, separate from the test-time verifier. SpatialVID and AgiBot provide captions, and DL3DV-10K and RealEstate10K use annotated scene descriptions. Other datasets train without text conditioning.
| Dataset | RGB | Depth | Camera pose |
|---|---|---|---|
| RealEstate10K | Yes | — | Yes |
| DL3DV-10K | Yes | VIPE | VIPE |
| SpatialVID | Yes | — | Yes |
| ScanNet++ | Yes | VIPE | VIPE |
| AgiBot World | Yes | — | — |
| OmniWorld | Yes | Yes | Yes |
| Virtual KITTI 2 | Yes | Yes | Yes |
| PointOdyssey | Yes | Yes | Yes |
Handling several tasks in one network is useful, but it does not by itself show that they help each other. We test this with controlled ablations: the same clips, 7,500 optimizer steps, and a matched training budget, removing one task or design choice at a time.
Removing RGB generation raises KITTI depth error from 0.152 to 0.322. Removing depth prediction raises ScanNetV2 rotation error from 2.91° to 4.30°. Removing pose estimation lowers RealEstate10K generation PSNR from 20.18 to 13.26 dB. Each task contributes to a different task’s performance.
We also remove RealEstate10K, which provides camera supervision but no depth labels. Depth error still rises on Bonn, from 0.135 to 0.239, and on KITTI, from 0.152 to 0.196. This is evidence that depth prediction can benefit from data that does not directly supervise depth.
The architecture matters too. Replacing image-aligned ray maps with global camera tokens worsens pose estimation and generation in the reported ablations. A single shared transformer also loses performance on several tasks, although it is slightly better for Bonn depth. The benefit comes from allowing the modalities to interact while retaining their own representations.
Each comparison uses the same sampled clips and hyperparameters, with an effective batch size of 8. Removing a task sets its loss to zero and replaces its target inputs with noise; retained loss weights are not renormalized. When RGB generation is removed, RGB remains available as clean conditioning. These are controlled smaller-budget runs, separate from the main-model benchmark tables.
| Evaluation | Complete architecture | Removed task or data |
|---|---|---|
| KITTI depth Abs Rel ↓; without RGB generation | 0.152 | 0.322 |
| ScanNetV2 pose RPEr ↓ (°); without depth prediction | 2.91 | 4.30 |
| RealEstate10K generation PSNR ↑; without pose estimation | 20.18 | 13.26 |
| Bonn depth Abs Rel ↓; without RealEstate10K | 0.135 | 0.239 |
| KITTI depth Abs Rel ↓; without RealEstate10K | 0.152 | 0.196 |
Given a camera trajectory and either the first image or the first and last images, UGG generates the remaining video. We compare quantitatively with Aether, another unified generation-and-geometry model
On RealEstate10K, SpatialVID, and DL3DV, UGG leads the compared methods on all three reported metrics in the one-view setting: PSNR and SSIM measure image agreement, while LPIPS measures perceptual similarity. With two views, it leads on RealEstate10K and SpatialVID; SEVA is stronger on DL3DV. For example, two-view RealEstate10K PSNR is 25.73 dB, compared with 25.25 for SEVA and 20.02 for Aether.
The videos also let us inspect something a table cannot fully describe: whether dynamic content remains coherent as the viewpoint changes. A requested camera motion and a moving person both have to be accounted for in the same completion.
Both protocols also receive the target camera trajectory. Wan2.1-FLF and DFoT are qualitative comparisons, not rows in the current quantitative table. Best and second-best shading is computed among the non-oracle rows within each protocol, with ties determined by the displayed rounded values. Oracle rows are ground-truth diagnostics and excluded from that ranking.
| Method | RealEstate10K | SpatialVID | DL3DV | ||||||
|---|---|---|---|---|---|---|---|---|---|
| PSNR ↑ | LPIPS ↓ | SSIM ↑ | PSNR ↑ | LPIPS ↓ | SSIM ↑ | PSNR ↑ | LPIPS ↓ | SSIM ↑ | |
| Aether | 20.02 | 0.196 | 0.624 | 13.84 | 0.489 | 0.334 | 13.14 | 0.399 | 0.243 |
| SEVA | 25.25 | 0.081 | 0.838 | 15.44 | 0.344 | 0.466 | 17.61 | 0.211 | 0.552 |
| UGG | 25.73 | 0.061 | 0.849 | 16.99 | 0.288 | 0.535 | 16.61 | 0.239 | 0.495 |
| Method | RealEstate10K | SpatialVID | DL3DV | ||||||
|---|---|---|---|---|---|---|---|---|---|
| PSNR ↑ | LPIPS ↓ | SSIM ↑ | PSNR ↑ | LPIPS ↓ | SSIM ↑ | PSNR ↑ | LPIPS ↓ | SSIM ↑ | |
| Aether | 15.79 | 0.335 | 0.469 | 11.81 | 0.632 | 0.283 | 11.96 | 0.506 | 0.240 |
| SEVA | 19.58 | 0.194 | 0.703 | 13.35 | 0.472 | 0.343 | 13.50 | 0.393 | 0.335 |
| UGG | 21.79 | 0.150 | 0.779 | 14.86 | 0.428 | 0.405 | 13.92 | 0.385 | 0.346 |
For geometric prediction, we distinguish dense inputs, where the full RGB video is observed, from sparse inputs, where only the first and last images are given to the model. We compare pose quantitatively with Geo4D, Gen3R, and Aether
Dense pose is evaluated on RealEstate10K, Sintel, TUM, ScanNetV2, and ScanNet++. After aligning trajectories up to global rotation, translation, and scale, we measure absolute trajectory error (ATE) and relative translation and rotation errors. Without search, UGG has lower ATE and relative translation error than all three external baselines on all five benchmarks. On ScanNet++, ATE is 0.087, compared with 0.190 for Geo4D. Rotation is also strong, though Gen3R remains better on TUM.
With only two observed images, we evaluate their relative pose. Translation angular error (tAE) measures whether the predicted direction of motion is correct; rotation error measures the change in orientation. UGG has the lowest errors among the external baselines on both metrics across all five datasets. On RealEstate10K, its translation angular error is 8.260°, compared with 39.834° for Aether, 44.004° for Gen3R, and 117.079° for Geo4D.
Dense trajectories use Sim(3) Umeyama alignment. ATE and RPEt are reported in the paper’s aligned coordinate convention; RPEr is in degrees. Sparse pose scores the relative camera between the two observed endpoints, with both tAE and RPEr in degrees. UGG uses shared intrinsics. Sparse TTS uses additional real RGB for verification, as described in Section 6; the base UGG and external baseline rows use the sparse model-input protocol. Best and second-best shading is computed among the non-oracle rows within each protocol, with ties determined by the displayed rounded values. Oracle rows are ground-truth diagnostics and excluded from that ranking.
| Method | RealEstate10K | Sintel | TUM | ScanNetV2 | ScanNet++ | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ATE ↓ | RPEt ↓ | RPEr ↓ (°) | ATE ↓ | RPEt ↓ | RPEr ↓ (°) | ATE ↓ | RPEt ↓ | RPEr ↓ (°) | ATE ↓ | RPEt ↓ | RPEr ↓ (°) | ATE ↓ | RPEt ↓ | RPEr ↓ (°) | |
| Geo4D | 0.026 | 0.028 | 0.147 | 0.160 | 0.097 | 0.911 | 0.052 | 0.029 | 1.414 | 0.074 | 0.044 | 1.762 | 0.190 | 0.090 | 5.591 |
| Gen3R | 0.030 | 0.011 | 0.141 | 0.233 | 0.066 | 1.538 | 0.046 | 0.016 | 0.943 | 0.075 | 0.024 | 1.018 | 0.232 | 0.066 | 5.760 |
| Aether | 0.018 | 0.014 | 0.851 | 0.180 | 0.867 | 1.354 | 0.041 | 0.016 | 1.462 | 0.075 | 0.030 | 1.617 | 0.225 | 0.068 | 5.944 |
| UGG | 0.010 | 0.004 | 0.076 | 0.098 | 0.034 | 0.708 | 0.018 | 0.007 | 1.052 | 0.038 | 0.016 | 0.938 | 0.087 | 0.036 | 3.861 |
| UGG (TTS) | 0.009 | 0.004 | 0.076 | 0.091 | 0.033 | 0.684 | 0.017 | 0.007 | 1.054 | 0.038 | 0.016 | 0.937 | 0.082 | 0.035 | 4.105 |
| UGG (Oracle; GT diagnostic) | 0.008 | 0.004 | 0.076 | 0.072 | 0.030 | 0.695 | 0.015 | 0.006 | 1.053 | 0.033 | 0.015 | 0.939 | 0.072 | 0.035 | 4.078 |
| Method | RealEstate10K | Sintel | TUM | ScanNetV2 | ScanNet++ | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| tAE ↓ (°) | RPEr ↓ (°) | tAE ↓ (°) | RPEr ↓ (°) | tAE ↓ (°) | RPEr ↓ (°) | tAE ↓ (°) | RPEr ↓ (°) | tAE ↓ (°) | RPEr ↓ (°) | |
| Geo4D | 117.079 | 3.407 | 108.628 | 24.160 | 100.224 | 11.936 | 90.812 | 43.458 | 82.747 | 83.019 |
| Gen3R | 44.004 | 4.197 | 75.984 | 24.984 | 82.875 | 12.374 | 79.871 | 43.481 | 92.324 | 82.540 |
| Aether | 39.834 | 14.969 | 78.012 | 33.640 | 48.436 | 6.234 | 70.925 | 36.720 | 77.234 | 81.681 |
| UGG | 8.260 | 1.287 | 35.479 | 22.065 | 9.938 | 1.371 | 28.240 | 21.417 | 40.194 | 43.078 |
| UGG (TTS) | 3.475 | 1.393 | 18.733 | 20.646 | 8.520 | 1.444 | 20.057 | 18.911 | 31.835 | 37.875 |
| UGG (Oracle; GT diagnostic) | 1.550 | 1.399 | 8.521 | 26.135 | 3.401 | 1.154 | 9.683 | 20.688 | 7.892 | 43.836 |
We evaluate depth on ScanNet++, 7-Scenes, KITTI, Sintel, and Bonn, against ChronoDepth, Geo4D, Gen3R, Aether, and Marigold
UGG is competitive with specialized depth systems across these settings. In dense ScanNet++, its Abs Rel is 0.073, compared with 0.106 for Geo4D, the strongest external baseline there. Its advantage is particularly clear with sparse observations: UGG leads the external baselines on both depth metrics on ScanNet++, 7-Scenes, Sintel, and Bonn. Specialized models retain advantages on some dense benchmarks, including Aether on KITTI and Geo4D on Sintel.
Depth is aligned for scale and shift at the video level. Abs Rel is lower-is-better and δ1.25 is higher-is-better. Sparse depth evaluates the two observed endpoints. Sparse TTS additionally reads neighbouring real RGB during selection, as described in Section 6. Oracle metrics describe a ground-truth-selected candidate, not an independently optimized best value in each column. Best and second-best shading is computed among the non-oracle rows within each protocol, with ties determined by the displayed rounded values. Oracle rows are ground-truth diagnostics and excluded from that ranking.
| Method | ScanNet++ | 7-Scenes | KITTI | Sintel | Bonn | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| Abs Rel ↓ | δ1.25 ↑ | Abs Rel ↓ | δ1.25 ↑ | Abs Rel ↓ | δ1.25 ↑ | Abs Rel ↓ | δ1.25 ↑ | Abs Rel ↓ | δ1.25 ↑ | |
| ChronoDepth | 0.177 | 0.724 | 0.116 | 0.871 | 0.120 | 0.857 | 0.394 | 0.530 | 0.092 | 0.920 |
| Geo4D | 0.106 | 0.894 | 0.080 | 0.923 | 0.097 | 0.923 | 0.247 | 0.676 | 0.057 | 0.972 |
| Gen3R | 0.157 | 0.772 | 0.079 | 0.937 | 0.185 | 0.777 | 0.414 | 0.520 | 0.168 | 0.805 |
| Aether | 0.166 | 0.769 | 0.109 | 0.898 | 0.056 | 0.975 | 0.351 | 0.607 | 0.095 | 0.911 |
| Marigold | 0.175 | 0.730 | 0.094 | 0.910 | 0.098 | 0.910 | 0.388 | 0.517 | 0.082 | 0.933 |
| UGG | 0.073 | 0.952 | 0.083 | 0.921 | 0.064 | 0.960 | 0.303 | 0.654 | 0.060 | 0.963 |
| UGG (TTS) | 0.072 | 0.954 | 0.084 | 0.920 | 0.063 | 0.962 | 0.293 | 0.661 | 0.059 | 0.969 |
| UGG (Oracle; GT diagnostic) | 0.063 | 0.963 | 0.074 | 0.923 | 0.055 | 0.967 | 0.263 | 0.692 | 0.056 | 0.973 |
| Method | ScanNet++ | 7-Scenes | KITTI | Sintel | Bonn | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| Abs Rel ↓ | δ1.25 ↑ | Abs Rel ↓ | δ1.25 ↑ | Abs Rel ↓ | δ1.25 ↑ | Abs Rel ↓ | δ1.25 ↑ | Abs Rel ↓ | δ1.25 ↑ | |
| ChronoDepth | 0.199 | 0.700 | 0.130 | 0.844 | 0.089 | 0.905 | 0.424 | 0.487 | 0.072 | 0.960 |
| Geo4D | 0.177 | 0.737 | 0.098 | 0.912 | 0.100 | 0.897 | 0.310 | 0.554 | 0.067 | 0.966 |
| Gen3R | 0.238 | 0.620 | 0.124 | 0.867 | 0.104 | 0.888 | 0.347 | 0.510 | 0.127 | 0.847 |
| Aether | 0.269 | 0.567 | 0.167 | 0.772 | 0.149 | 0.786 | 0.487 | 0.424 | 0.138 | 0.849 |
| Marigold | 0.169 | 0.733 | 0.106 | 0.890 | 0.087 | 0.921 | 0.345 | 0.527 | 0.071 | 0.970 |
| UGG | 0.101 | 0.884 | 0.087 | 0.916 | 0.099 | 0.917 | 0.258 | 0.660 | 0.049 | 0.977 |
| UGG (TTS) | 0.082 | 0.917 | 0.086 | 0.916 | 0.087 | 0.934 | 0.219 | 0.678 | 0.047 | 0.979 |
| UGG (Oracle; GT diagnostic) | 0.055 | 0.966 | 0.075 | 0.923 | 0.068 | 0.946 | 0.177 | 0.781 | 0.043 | 0.979 |
Sparse observations can admit several plausible scene explanations. A generative model can represent alternatives, so we can spend additional inference compute drawing candidates from different noise initializations. In our experiments, we sample 16 candidates and select one for each scene.
The difficulty is deciding which candidate to trust without the correct depth or camera pose. We use a verifier built around RAFT optical flow
First, the verifier checks flow in both directions and removes correspondences whose round trip is inconsistent. It also adapts the spacing between frames to the measured motion. Adjacent frames can have too little parallax to constrain depth; widely separated frames can have too much occlusion for reliable matching. Choosing pairs by pixel displacement gives the geometric check more useful evidence.
Checking depth. Camera rotation moves pixels even for very distant surfaces. Translation adds parallax, whose magnitude depends on depth. We estimate a relative camera from the image correspondences and use that same camera for every depth candidate. After accounting for rotation and fitting the candidate’s inverse-depth scale and shift, we ask how well its depth explains the remaining motion. Using a shared camera helps separate errors in depth from errors in predicted camera pose.
Checking pose. For each candidate, we compare its predicted relative translation direction with the direction estimated from the images, using that candidate’s intrinsics. The selected pose rule averages these angular disagreements over several frame pairs. It evaluates each candidate against image evidence, rather than rewarding agreement with the other predictions.
In the sparse setting, the model receives the two endpoint images, while the verifier also uses real neighbouring RGB frames from the clip. Depth is checked at the scored endpoints against nearby images; pose is checked along the candidate’s predicted trajectory. This lets us generate hypotheses from sparse inputs and use additional visual evidence to choose among them, without requiring depth or pose labels.
| Task | Evidence and selected readout | Frame pairs |
|---|---|---|
| Sparse depth | Parallax fit residual + perpendicular epipolar residual (corr + epi) | 8 endpoint-to-neighbour pairs; 16 px motion target |
| Dense depth | Parallax fit residual + DINOv2 feature warp residual + RGB warp residual (corr + feat + photo) | 12 adaptive pairs; 16 px target |
| Sparse pose | Translation-direction angle (t) | 12 adaptive pairs along the predicted trajectory; 16 px target |
| Dense pose | Translation-direction angle (t) | 32 directed pairs from 16 bidirectional anchors; 32 px target |
The verifier selects one complete candidate for each scene. Depth scores are combined through weighted ranks across frame pairs; pose uses the ordering of its translation-direction score. Each candidate is checked against the images rather than a consensus prediction from the pool. We assess the choices of readouts and pair recipes by leaving one benchmark out, choosing on the others, and evaluating on the held-out benchmark. The sparse rules are the same in every fold.
The largest gains appear in sparse prediction. RealEstate10K tAE falls from 8.260° to 3.475°, and Sintel tAE from 35.479° to 18.733°. Sparse ScanNet++ depth Abs Rel falls from 0.101 to 0.082; sparse Sintel falls from 0.258 to 0.219. Dense gains are generally smaller. The verifier is a proxy for geometric accuracy, so improvements are not uniform across every metric, but the sparse results show that the model generates useful alternatives and that image evidence can help identify better ones.
The tables also include an oracle diagnostic: ground-truth-based selection from the same candidate pool. The gap between the verifier and oracle shows room to make better use of the model’s candidate diversity. This gives us two directions for improving prediction: learn a stronger scene distribution, and develop better ways to check its samples.
Correspondences and pair spacing. RAFT-Large is run in both directions. A correspondence from $x$ to $x’=x+F_{ij}(x)$ is retained only if $x’$ is in bounds and
\[\|F_{ij}(x)+F_{ji}(x')\|^2\leq 0.01\big(\|F_{ij}(x)\|^2+\|F_{ji}(x')\|^2\big)+0.5.\]The consistency check filters unreliable correspondences before geometric fitting. For an anchor $i$, the gap is $\operatorname{clip}(\operatorname{round}(\tau/\operatorname{median}x|F{i,i+1}(x)|),1,12)$. The target $\tau$ is 16 px except for dense pose, which uses 32 px. It is shared across benchmarks within each protocol. Sparse depth pairs each endpoint with real interior neighbours at 1, 1.5, 2, and 2.5 times its adaptive gap. The verifier processes images at 288 × 384 and evaluates residuals on a stride-2 pixel grid.
Depth evidence. A five-point essential-matrix estimate uses up to 6,000 consistent correspondences, RANSAC probability 0.999, and a 1 px threshold, followed by pose recovery. The resulting unit-translation camera is shared across depth candidates. In the current implementation, this estimate is cached using the intrinsics of the first candidate processed. Candidate depths are normalized in inverse depth, and a two-parameter affine gauge is fitted by texture-weighted least squares. Pixels require more than 0.5 px of predicted parallax; the fit retains pixels within the 80th percentile of the perpendicular parallax residual. These filters reduce contamination from motion not explained by the camera model. The corr readout measures the texture-weighted along-parallax discrepancy; epi measures the texture-weighted perpendicular residual over valid pixels. photo is the mean absolute RGB difference under the gauged warp, and feat is mean cosine distance between DINOv2 ViT-S/14 features under that warp.
Depth aggregation. Within each scene and pair, each named readout is z-normalized across candidates. The normalized readouts are combined with equal weight, then candidates are ranked within the pair. Pair weights use the interquartile spread across candidates (disc); pairs that cannot distinguish candidates carry little weight. Weighted mean rank gives the final candidate score. This uses pool statistics for normalization, discrimination weighting, and ranking, but does not use a consensus depth or camera as geometric evidence.
Pose evidence and aggregation. For candidate camera-to-world transforms, the relative camera from $i$ to $j$ is $T_p=\mathrm{c2w}_j^{-1}\mathrm{c2w}_i$. The image-derived relative pose is estimated with that candidate’s own intrinsics. Translation angle t, rotation angle r, and the median Sampson residual samp were considered; the selected rules use only t. Pair angles are averaged per candidate, then candidates are ranked. With a single readout, ranking preserves its ordering. Sparse trajectory pairs without a saved candidate camera are skipped; the method does not invent missing poses.
Rule selection. Pair recipes, readout combinations, and motion targets were assessed using leave-one-benchmark-out selection within each task and protocol. The held-out benchmark is excluded when choosing its rule. Sparse depth and sparse pose select the same rule in every fold. The table above gives the final task-level rules. Ground-truth metrics are used for this offline rule-selection study and for evaluation, not as inputs when scoring a new scene.
Abstention. If no usable pair remains for a scene, the verifier abstains. Its reported performance for that scene is the expected metric of a random draw from the candidate pool, rather than a ground-truth-based fallback choice.
The results suggest that generation and geometric prediction can reinforce each other when they are learned as parts of the same scene model. Geometry helps constrain visual generation. Learning appearance and camera motion can improve depth, including through training examples without depth labels. A generative formulation also provides alternative predictions that can be checked at inference time.
The current model operates on short clips at modest resolution, and repeated sampling adds inference cost. Specialized systems remain stronger in some settings, and the gains from search depend on the quality of the visual evidence. These are useful directions for extending the approach to longer videos, larger models, and more effective verification.
A useful next step is to improve how candidates are checked, and eventually use that evidence during denoising rather than only after a full completion. The central idea is already useful: appearance, depth, and camera motion can be learned together, with the observed parts of a scene determining which question the model answers.
This post accompanies UGG: Unified Generative Geometry Model, by Amani Kiruga, Zhiyi Li, Ruojin Cai, Hansen Lillemark, Xiaoshan Wu, Ayush Tewari, and Yilun Du. The project used computing resources at the Kempner Institute at Harvard University.