Volumetric Disentanglement for 3D Scene Manipulation

Sagie Benaim*, Frederik Warburg, Peter Ebert Christensen, Serge Belongie

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

Recently, advances in differential volumetric rendering enabled significant breakthroughs in the photo-realistic and fine-detailed reconstruction of complex 3D scenes, which is key for many virtual reality applications. However, in the context of augmented reality, one may also wish to effect semantic manipulations or augmentations of objects within a scene. To this end, we propose a volumetric framework for (i) disentangling or separating, the volumetric representation of a given foreground object from the background, and (ii) semantically manipulating the foreground object, as well as the background. Our framework takes as input a set of 2D masks specifying the desired foreground object for training views, together with the associated 2D views and poses, and produces a foreground-background disentanglement that respects the surrounding illumination, reflections, and partial occlusions, which can be applied to both training and novel views. Our method enables the separate control of pixel color and depth as well as 3D similarity transformations of both the foreground and background objects. We subsequently demonstrate our framework's applicability on several downstream manipulation tasks, going beyond the placement and movement of foreground objects. These tasks include object camouflage, non-negative 3D object in-painting, 3D object translation, 3D object inpainting, and 3D text-based object manipulation. The project webpage is provided in https://sagiebenaim.github.io/volumetric-disentanglement/.

Original languageEnglish
Title of host publicationProceedings - 2024 IEEE Winter Conference on Applications of Computer Vision, WACV 2024
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages8652-8662
Number of pages11
ISBN (Electronic)9798350318920
DOIs
StatePublished - 3 Jan 2024
Externally publishedYes
Event2024 IEEE Winter Conference on Applications of Computer Vision, WACV 2024 - Waikoloa, United States
Duration: 4 Jan 20248 Jan 2024

Publication series

NameProceedings - 2024 IEEE Winter Conference on Applications of Computer Vision, WACV 2024

Conference

Conference2024 IEEE Winter Conference on Applications of Computer Vision, WACV 2024
Country/TerritoryUnited States
CityWaikoloa
Period4/01/248/01/24

Bibliographical note

Publisher Copyright:
© 2024 IEEE.

Keywords

  • 3D computer vision
  • Algorithms
  • Applications
  • Virtual / augmented reality

Fingerprint

Dive into the research topics of 'Volumetric Disentanglement for 3D Scene Manipulation'. Together they form a unique fingerprint.

Cite this