Thanks to volumetric video technology, it is possible to capture people and objects through a variety of cameras and digitise them in the form of models. As experts in this field of 3D reconstruction and motion capturing, we have worked together with the Fraunhofer Heinrich-Herz-Institut and the agency Reallifefilm International to make changes to models in post-production possible. The field of application for this technology is incredibly versatile and continues to expand as the hardware advances. Realistic representations and movements can already be inserted into digital worlds and even viewed in VR or AR.

The basic idea behind volumetric video was born several decades ago in the field of sports in particular to take pictures from different angles. If several cameras are already aimed at one athlete, why not try to get a 3-dimensional image from these shots? The idea was then implemented for the first time in photogrammetry, as in Google Street View. An object is photographed from multiple sides and then displayed in 3D here. But there was a lack of powerful hardware for a long time when it came to recording video. Not only has VR made great progress in recent years, overall computing power continues to improve..

A volumetric video is created by the use of numerous cameras, which together represent a picture from different angles that is as accurate and realistic as possible. There are no limits to the use of the picture material. The pictures offer incredibly detailed quality both for movies and games. Digital animation doesn’t come close to the accuracy of the depth cameras here because the image digitally tries to look as natural as possible. In addition, animation requires several hours of work for one minute of video, while volumetric video directly records the natural movements and transmits them digitally.

Now you might be thinking: “That sounds pretty good, but what if I want to change something later?” That’s where we come into play. In collaboration with the Frauenhofer HHI and RealLifeFilm we developed the “Sorcerer’s Apprentice” as part of VVOW . Actors were recorded in a studio using 32 cameras, then subsequently reconstructed and modelled in 3D for this purpose. After the recordings were done and the texturing in RGB-data had taken place, it was now possible for us to alter the model in the subsequent steps. This post-production process makes it possible for the user to interact with volumetrically recorded figures.

As if this feature were not revolutionary enough, imagine that you could experience it in VR. The characters are not only lifelike; they also interact naturally with you. Both the user experience and the storytelling is unbelievably upgraded and feels like an evolved form of realism. The player feels more like part of their own story and isn’t just enchanted by beautiful animations. Of course, it’s possible to try to create digital images and render them in a more and more realistic way, but what could be more realistic than images of the original?

In addition to years of experience in storytelling and technology using volumetric videos, we have been able to further cut down on the effort required. Thanks to the development of different software and different players, we have been able to significantly reduce problems of high memory usage. This makes working with models a lot easier and much more user-friendly. We are also developing a mobile device player to take full advantage of the versatility of immersive media.

There are no limits to the future

The potential for volumetric video seems limitless. And if you start to think about the future, you’ll come across some interesting new applications. Depth camera technology is already well advanced and arrived in German living rooms with the Xbox Kinect. The performance of the hardware is also expanding steadily and will soon overshadow the cost of the modelling studio. We are already living in a world of wonderful details, which thanks to VR is making dreams a reality.