NeurIPS 2024

Temporally Consistent Atmospheric Turbulence Mitigation

with Neural Representations

Haoming Cai1,*   Jingxi Chen1,*   Brandon Y. Feng2   Weiyun Jiang3   Mingyang Xie1   Kevin Zhang1   Cornelia Fermüller1   Yiannis Aloimonos1   Ashok Veeraraghavan3   Christopher A. Metzler1

1University of Maryland  ·  2Massachusetts Institute of Technology  ·  3Rice University  ·  *Equal contribution

TL;DR A representation with only a few temporal features can’t express fast jitter: a low-pass filter over time.

Restored video still flickers. Why
Limit the time resolution. See it
On top of existing ATM methods. Results
Paper NeurIPS page BibTeX
Input vs. ConVRT, then slices of the fin through time.
1 · Motivation2 · Insight3 · Method4 · Results

Restored turbulent video still flickers

Video turbulence-mitigation (ATM) methods clean up each frame, but the result is not consistent from one frame to the next.

Crops of a static pole at three times t0, t1, t2 for the original video, VRT, TMT, DATUM and DATUM plus ours
A static pole flickers after state-of-the-art video ATM methods (crops at t0, t1, t2). Paper Fig. 1.
1 · Motivation2 · Insight3 · Method4 · Results

Limited temporal capacity is the low-pass filter

Over time, turbulence is high-frequency jitter, like noise. ConVRT gives its video representation only a few temporal features (Tres ≪ T; Tres = 5 in the paper), so it can only learn the low-frequency part: the jitter is removed and the slow motion stays.

Real frames and y–t slices of the aircraft clip below (the clip’s own marker lines are removed or dimmed). The edge path with fewer temporal features is fitted to the measured edge, as an illustration.

Zoomed y–t slices on a synthetic video: GT, LQ, TurbNet, plus ours, T_res equals 8, without L_temp
Paper Fig. 7 (synthetic video, zoomed y–t slices): residual turbulence goes with a lower Tres and Ltemp.
1 · Motivation2 · Insight3 · Method4 · Results

Two fields, one of them regularized

A Temporal Deformation Field predicts how far each pixel is shifted from a canonical frame, and a Spatial Content Field reads its colour there. ConVRT is fitted to each video, guided by the output of an existing ATM method.

Diagram of the Temporal Deformation Field and the Spatial Content Field
The regularization: a Temporal Feature Map with low time resolution Tres and a small deformation MLP. Paper Fig. 4.
1 · Motivation2 · Insight3 · Method4 · Results

Real-world turbulence

Added to VRT, TMT or DATUM, ConVRT steadies feature tracks and straightens y–t slices.

Three real-world scenes: KLT tracks and y–t slices for VRT, TMT and DATUM, each with and without ours
KLT tracks (top) and y–t slices (bottom), each method with and without ConVRT. Paper Fig. 5.

More results

Aircraft tail. Base methods with and without ConVRT (“+ Ours”).
Target in a field.
A moving subject.
Resolution chart.
Four base methods. TurbNet, DATUM, TMT and VRT, each with and without ConVRT.

After the argument · Citation

BibTeX

@article{cai2024temporally, title={Temporally consistent atmospheric turbulence mitigation with neural representations}, author={Cai, Haoming and Chen, Jingxi and Feng, Brandon Y and Jiang, Weiyun and Xie, Mingyang and Zhang, Kevin and Fermuller, Cornelia and Aloimonos, Yiannis and Veeraraghavan, Ashok and Metzler, Christopher A}, journal={Advances in Neural Information Processing Systems}, volume={37}, pages={44554--44574}, year={2024} }