Flow-SDF —
Rectified Flow for Neural SDF Reconstruction.

Can a flow model replace the CNN encoder — and keep "no 3-D supervision"?

Mini SDF-SRN (Topic 43) proved you can learn 3-D from single-view silhouettes alone — but its CNN encoder is deterministic, mapping one image to one code in a single forward pass. That is a dead end for generation: no stochasticity, no conditioning on text or partial views, no iterative refinement. Modern image-generation systems (Stable Diffusion 3) and state-of-the-art 3-D systems (Hunyuan3D 2.0, TRELLIS 2) all replaced deterministic encoders with iterative denoising — flow-based diffusion transformers conditioned on image features.

Flow-SDF asks the precise question: can we swap the CNN encoder for a rectified-flow model while keeping the one property that matters — requiring no 3-D supervision? The answer is yes. The conventional systems build their 3-D latent space in a separate first stage, trained on millions of 3-D meshes; Flow-SDF builds it jointly, from 2-D images only, with the SDF decoder and the flow co-evolving through the differentiable renderer.

Image → conditioner → rectified flow (8 steps) → SDF decoder → renderer.

Four components, ~7 M parameters total, all trained end-to-end.

The rectified flow uses AdaLN — the same conditioning technique as in the Diffusion Transformer — to modulate layer-norm parameters by the diffusion timestep, so the network behaves differently at each stage of denoising. At inference, starting from scaled Gaussian noise, the velocity field is integrated with 8 Euler steps to produce a clean SDF latent code.

Flow distillation, then end-to-end fine-tuning.

Phase 1 — flow distillation. A standard CNN-based SDF-SRN model is trained to convergence first (this is Mini SDF-SRN, Topic 43). Its trained encoder generates target latent codes for all training images; the rectified flow is then trained to reproduce those codes with two complementary losses — a velocity loss (MSE between predicted and target velocity at random timesteps, training local accuracy) and a sampling loss (MSE between the full 8-step Euler-integrated output and the target, training end-to-end integration quality). Phase 1 is fast — ~30 seconds total, no rendering involved.

Phase 2 — end-to-end fine-tuning. All parameters unfrozen. The full pipeline runs image → conditioner → flow (8 steps) → latent → SDF decoder → differentiable renderer → silhouette, with binary-cross-entropy loss against the ground-truth silhouette. Phase 2 is slow (~14 s/epoch) because each step requires 8 flow denoising passes followed by SDF evaluation at 196,608 spatial points (64 × 64 pixels × 48 depth samples). 200 epochs brings the silhouette loss from ~1.5 down to 0.0053.

Three stages — silhouette baseline, flow + silhouette, flow + RGB.

The project runs in three stages. Stage 1 is the CNN-encoder silhouette baseline — the Mini SDF-SRN reimplementation (Topic 43). Stage 2 replaces the CNN encoder with the 8-step rectified flow. Stage 3 adds differentiable RGB rendering with Lambertian shading on top. The figures below are the actual epoch outputs from the repository.

Stage 2 — rectified flow + silhouette

Sphere · Phase 2, epoch 200. Training view + 4 novel angles. Pred row vs GT row — the 8-step flow matches the CNN baseline's silhouette quality.

Box · Phase 2, epoch 200. The hardest case — sharp corners. The flow recovers the gross box silhouette across all 4 novel angles.

Ellipsoid · Phase 2, epoch 200. The anisotropic silhouette is recovered at every novel angle.

Stage 2 training convergence · Phase 2, epoch 200. Ground-truth / predicted / difference. The difference shrinks to a thin boundary band — the rectified flow reaches the same reconstruction quality as the CNN encoder.

Stage 3 — rectified flow + RGB

Stage 3 adds a differentiable RGB renderer with Lambertian shading. The intuition: a sphere and a cube have similar silhouettes but very different shading patterns — RGB supervision provides dense surface- normal information that silhouette-only supervision cannot.

Sphere · RGB Flow-SDF, epoch 100. Three rows — predicted RGB, predicted silhouette, ground-truth RGB. Lambertian shading gradients add surface-normal supervision on top of the boundary signal.

Box · RGB Flow-SDF, epoch 100. The shading distinguishes a box from a sphere even where the silhouettes are similar.

Ellipsoid · RGB Flow-SDF, epoch 100. RGB + silhouette combined supervision.

Stage 3 RGB training convergence · epoch 100. The RGB rendering loss converges alongside the silhouette loss; shading gradients provide the dense surface-normal signal.

Gradient-flow validation: all three trainable components receive non-zero gradients when loss is backpropagated through the full chain — conditioner (grad norm 3.91), velocity network (161.93), SDF decoder (324.63). The forward pass completes in 258 ms, the backward pass in 282 ms on an Apple M4.

Speed for capacity; 2-D-only for geometric precision.

Speed vs capacity. The CNN encoder produces a code in one ~2 ms forward pass; the rectified flow needs 8 denoising steps (~20 ms), and training is ~3× slower per epoch. But the flow architecture supports stochastic sampling, conditioning on arbitrary modalities, and iterative refinement — capabilities the CNN encoder fundamentally lacks. 3-D supervision vs 2-D-only. The 2-D silhouette signal is geometrically weaker than direct 3-D supervision — it constrains the visual hull but leaves depth ambiguity within it. The shared decoder resolves this statistically across the dataset, but the geometry is less precise than 3-D-supervised methods. The trade is deliberate: zero 3-D data is a significant practical advantage where 3-D assets are scarce.

The scaling path the report lays out is concrete: replace the CNN conditioner with a frozen DINOv2-Small (background- invariant features, no training); replace the MLP velocity network with a small transformer DiT using cross-attention between latent tokens and image tokens; combine silhouette + RGB supervision; add a CLIP text encoder for text-to-3-D; and extend to compositional shapes with spatially-structured latents (3-D grids or triplanes). Each is a modular, independently-testable substitution.