Using an array of high-resolution cameras, scientists at Fraunhofer Heinrich Hertz Institute (HHI) in Berlin, Germany have developed a state-of-the-art scanning technique to capture realistic images of a person and transmit them to a virtual reality environment. The technology, which will be on display at IFA 2017 in Berlin, may make human actors in virtual and augmented reality experiences even more lifelike.
“Today you usually use animated avatars to model a person,” Ingo Feldmann, head of the Immersive Media and Communication group at Fraunhofer HHI, told Digital Trends. “So you create a model of a person in a 3D map and somehow capture the motion from a real actor and then you try to animate this model.”
The method developed by Feldmann and his team instead record high-resolution video using industrial stereo cameras, which capture an image with two lenses, much like a pair of human eyes. The array is made up of 32 of these cameras (Feldmann was tight-lipped about about the camera models) allowing it to record a subject’s movements in 360 degrees, creating a detailed and dynamic representation.
“We film the person directly in 3D,” Feldmann said, “in what we call volumetric video. There are cameras arranged around this actor … and these cameras create 2D images. From these images we create a 3D structure or model that looks completely natural. There are no animations, it’s the real person, and they can be inserted into a VR or AR experience.”
Of course, there’s a lot that goes into inserting the recorded video into a virtual world but, to make the task easier, the researchers created algorithms that can quickly extract the video’s depth data to calculate how the 3D figure should appear. Feldmann said the camera array is capable of capturing some 300 million triangles per person — far too many for a VR device to accommodate — so they must also reduce the complexity of the model.
“We have to adapt the resolution to the device,” he said.
The scientists have now opened a studio at Fraunhofer HHI where people can come to record models for VR and AR.
IFA 2016 was host to another state-of-the-art scanner, Botscan, which used 70 high-resolution DSLRs to capture an incredibly detailed, albeit static, 360-degree image.
