Parametric Shadow Control for Portrait Generation in Text-to-Image Diffusion Models

Haoming Cai1   Tsung-Wei Huang2   Shiv Gehlot2   Brandon Y. Feng3   Sachin Shah1   Guan-Ming Su2   Christopher Metzler1

1University of Maryland  ·  2Dolby Laboratories  ·  3MIT

Cheap synthetic data is enough to read out the shadows a frozen diffusion model already knows.

Code is not released because of copyright restrictions.

1 · Key idea2 · How it works3 · Comparisons

The diffusion model already knows the light

Simple data is enough

The training portraits are synthetic relights, far from photoreal. That is enough to read out the lighting the pretrained model already encodes.

Readout, not ControlNet

No new conditioning branch and no fine-tuning: a small estimator reads shadows from the frozen UNet's features, and generation is steered by optimizing only the latent.

1 · Key idea2 · How it works3 · Comparisons

How it works

Train two small estimators to read shadow, depth and identity from a frozen UNet, then use them to steer generation.

frozen trained / updated

    Drawn after the paper’s Fig. 2–5 and Appendix Fig. 12, with their own images. The portraits in “User input” and the final portrait are real outputs (paper Fig. 1; the final one is a different example from the maps shown during guidance). During guidance, ŝ converging to s* and the identity colour drifting and recovering are schematic.

    1 · Key idea2 · How it works3 · Comparisons

    Against other methods

    Relighting methods shift identity or miss the light; Shadow Director keeps the person and the style.

    Top row: Shadow Director controlling light direction, shadow strength and shadow shape. Bottom row: baseline methods on the same portraits.
    Ours (top) vs. baselines (bottom), which shift identity, miss the light, or barely change the shadow.
    Five relit versions of three portraits by Shadow Director, next to IC-Light, DiFaReli and the original.
    Ours (left five; inset: target shadow) vs. IC-Light and DiFaReli, with the original at right.
    Eight generated portraits relit from the left by Hou et al., DiFaReli, SwitchLight, IC-Light and Shadow Director, with the originals.
    Light from the left: one portrait per column, one method per row.

    Limitations

    Citation

    BibTeX

    @inproceedings{cai2025parametric, title = {Parametric shadow control for portrait generation in text-to-image diffusion models}, author = {Cai, Haoming and Huang, Tsung-Wei and Gehlot, Shiv and Feng, Brandon Y and Shah, Sachin and Su, Guan-Ming and Metzler, Christopher}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision}, pages = {18207--18217}, year = {2025} }