Google team relights avatars in any virtual environment
Presented during the 12th ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques held in Brisbane, the novel technique relies on a custom-designed full-body volumetric capture system whose output is fed to a hybrid geometric and machine learning reconstruction pipeline that processes the high resolution input and outputs a volumetric video complete with dynamic reflectance maps. This allows the researchers to „relight“ their avatar in any 3D rendered virtual environment while ensuring the subject lighting always remain consistent with its environment.
In a paper titled „The Relightables: Volumetric Performance Capture of Humans with Realistic Relighting“, the scientists demonstrate that their system enables a much higher resolution volumetric capture than previously achieved with other methods.
The main volumetric capture system consists in a custom light stage in the shape of a geodesic sphere integrating 331 custom programmable colour LEDs and a set of depth sensors based on active illumination, generating 60Hz per-viewpoint 12.4MP depth maps (4112×3008 pixels). For the actual volumetric capture, the system is also equipped with 58 RGB cameras, 32 IR cameras and 16 custom IR structured light projectors (each using a VCSEL diode through a chrome-on-glass direct-projection mask).
Key to „relighting“ the avatar is the „live reflectance map“ captured by extracting the information contained in two alternating color gradient illumination images acquired at 60Hz, creating reflectance maps for each 3D frame at a final output rate of 30Hz. Rather than rely on a green screen to contour-out the subject, the researchers used machine learning to compare frames (including images captured inside the empty geodesic sphere).
With various examples of virtual reality scenarios, the paper shows how the 3D avatar can then be realistically integrated into any lighting conditions.
https://augmentedperception.github.io/therelightables/
If you enjoyed this article, you will like the following ones: don't miss them by subscribing to :
