NVIDIA’s New AI: Erasing Reality

Why It Matters

Real‑time, zero‑training object removal lowers barriers for creators and could transform video production workflows.

Key Takeaways

• Omnimatte Zero removes objects and shadows from video in real time
• Technique leverages existing diffusion models, requiring zero additional training
• Uses temporal attention to copy background frames, causing slight blur
• Operates at 25 fps, enabling interactive video editing workflows
• Source code expected soon, promising open‑source community adoption

Summary

NVIDIA's research team unveiled Omnimatte Zero, a new AI system that can delete objects, shadows and other secondary effects from video footage in real time. The method, a collaboration between NVIDIA and external labs, builds on off‑the‑shelf diffusion models and requires no additional training, delivering 25 frames‑per‑second performance.

The core insight is temporal attention: the algorithm treats consecutive frames as a stack of jigsaw puzzles, copying exact background pieces from neighboring frames rather than hallucinating new pixels. This copy‑and‑paste approach eliminates the need for costly generative inference, but the averaging of multiple frames introduces a modest blur and occasional artifacts.

Demonstrations show puppies, a cat stepping on grass, and moving shadows being removed cleanly, with the presenter likening the process to a magnetic pull that extracts only the moving elements. He also noted that the source code is slated for release in early February, promising open‑source availability.

If widely adopted, Omnimatte Zero could democratize high‑quality video editing, enabling creators, advertisers and VFX studios to perform complex object removal instantly without specialized training data, potentially reshaping content‑creation pipelines.

Original Description

❤️ Check out Lambda here and sign up for their GPU Cloud: https://lambda.ai/papers

📝 The paper and the code are now available here:

https://dvirsamuel.github.io/omnimattezero.github.io/

https://github.com/dvirsamuel/OmnimatteZero

Our Patreon if you wish to support us: https://www.patreon.com/TwoMinutePapers

🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:

Adam Bridges, Benji Rabhan, B Shang, Cameron Navor, Christian Ahlin, Eric T, Fred R, Gordon Child, Juan Benet, Michael Tedder, Owen Skarpness, Richard Sundvall, Ryan Stankye, Steef, Taras Bobrovytsky, Tazaur Sagenclaw, Tybie Fitzhugh, Ueli Gallizzi

My research: https://cg.tuwien.ac.at/~zsolnai/

#nvidia