Being around someone using a virtual reality system can be an eerie experience. The typical VR headset fully hides the user’s eyes, making it hard to follow along with the action and interact with anyone using a system. When you’re inside a VR experience with another user, today’s systems don’t pass along facial expressions and so the social aspects and overall experience are hindered.
Google Research is looking to resolve that problem with a new project it is working on with Daydream Labs. By combining 3D computer vision, machine learning, and advanced rendering methods, the project aims to create more of a realistic solution that can convey a user’s identity, focus, and face.
Basically, the solution uses a VR headset modified by SensoMotoric Instruments that includes a video camera and eye-tracking components to show a person’s facial variations while blinking and looking in certain directions. This information is overlaid inside the mixed reality environment using green screen technology, basically making it look like the VR headset is a scuba mask — the translucent effect is deliberate and intended to avoid the “uncanny valley” effect that can make avatars and robots come across as unnatural.
From there, this modified user representation is worked into VR and mixed reality environments. The experienced is completed by showing a more realistic and expressive view of the user including their facial expressions and the ability to see where the user is looking.
While the technology does nothing for the real-world headset user, it dramatically improves the experience inside a VR or mixed reality world. Users can interact more realistically, using facial expressions and nonverbal cues to greatly enhance the communication and social interaction possible in VR environments. While the gaming applications are obvious, more mundane and practical solutions like VR video conferencing would also be greatly improved.
There is no word yet on when this technology will make it to shipping products. Anyone who wants to delve into the details can visit the Google Research blog to learn more about how the pieces connect to create a much more realistic virtual representation of the VR user.
