Learn Languages VR: Our first take

Bonjour, mon ami! Mondly uses VR to teach new languages

What if you could have a conversation with someone speaking in a different language, without being spoon fed individual words, whenever and wherever you want? This has long been the holy grail of language, and software has chased it for decades — with marginal success.

Yet there’s new hope to be found in virtual reaility, and Learn Languages VR by Mondly, the company’s first virtual reality application, shows why. In Mondly VR, you don’t interact with an instructor. Instead, you have conversations with a chat bot represented by a variety of digital characters, which asks you questions in the language you have selected.

During your conversations you are prompted with common responses to what the character says, and every answer you give is transcribed on screen, so you can see where you messed up. If you answer correctly, a green check mark will hover over the transcription of what you just said. Answer wrong, and the character will repeat the question. Not many human teachers would have this level of patience, but the chatbot is ready to repeat as often as needed.

Learn Languages VR doesn’t stop at chatbots. People learn best through real-world practice, and the application emulates that. You can choose from three VR experiences that test you knowledge — ordering at a restaurant, meeting someone on a train, and checking into a hotel.

So, can you really learn a new language in virtual reality? We gave it a go.

Robot Learning

Mondly is based on a server-side chat bot, which enables the app to speak with you without an actual person feeding it answers. Popular language learning program Rosetta Stone only teaches you new languages in conversations if you schedule a video chat session with one of its tutors. Being able to hear a voice recite a question often times made it easier to understand the pronunciation and context of how to use suggested phrases. And if you are planning to vacation in Barcelona, knowing how to say “can I get a cab” rather than simply a “cab” will certainly help you fit in.

The Mondly experience leverages the immersive quality of VR by the amount of interactivity you can access with just voice and head movements. You stare at the character in your simulation and as you speak, which is a much more natural environment than looking at a book or screen. With just the slightest head movement, you can view and access a transcription of what the chatbot said, see answer suggestions, view English translations, start audio recitations in the selected language, and more.

To get an English translations of suggestions and the chatbot’s answer, you simply hover the cursor over the phrase, and the translation will appear below. Hovering over the speaker icon near any phrase will prompt a voice to recite it back in the selected language. This helps pick up the nuances of languages predicated on vocal inflections, such as Japanese. But, sometimes the app accepts an answer that correctly pronounces a part of the phrase with a different inflection, if the app feels you got the gist of the response across.

Can You Hear Me Now?

The most important part of Mondly is not how intelligent it is, but how well it can hear. Learn Languages VR relies on the illusion of conversing with a person, and that means constant coordination between audio and visual components in order to make you feel like you are somewhere you actually are not. If you are learning a new language in VR, any noticeable lag or miscommunication between you and the instructor means the illusion is gone, which destroys the experience.

“The toughest challenge was to make the speech interaction feel natural in VR.”

“The toughest challenge was to make the speech interaction feel natural in VR,” Mondly co-founder Alexandru Iliescu wrote in a press release for the app. The primacy of sound to the experience is paramount. Before each experience you are prompted to remain quiet as Mondly’s voice detection system calibrates the microphone to the room’s background noise in order to ignore unintended sounds. I used Mondly in a silent office space and in a communal area with constant chatter around, and noticed no difference in the voice recognition.

Even so, the software has some issues to work out. Mondly’s voice recognition is consistently perfect…as long as it’s asked to handle phrases just a few words long. While trying to ask for a beer at a virtual restaurant in French, I could see only parts of my response transcribed.

Seeing the app wasn’t working, We decided to slowly say the sentence word-by-word to see if I was jumbling the pronunciation. When we did, the app displayed different words than what I was speaking, consistently. I yelled “Bier” in French 4 times, and was met “Heir.” Eventually, we gave up.

Too Smart To Teach

Mondly teaching you how to speak a different language through conversations is a double-edged sword. It is a crash course on context, verb tenses, and pronunciation in a language you probably never spoke before. The learning curve can be steep, and the app tends to throw users into the deep end before teaching them to swim.

We found a lesson might start with a simple phrase, then follow up by asking us to recite an eight word sentence introducing ourselves. This seems to make the app most useful to those who can read the language they wish to learn, but don’t know how to form sentences. Those Spanish classes you took in high school could come in handy.

Mondly VR

Each experience yields different levels of difficulty, as well. We talked through a light conversation on the train in German without much trouble. Trying to reserve a room with a shower was considerably more complex. After numerous failed attempts at complete phrases, we would repeat each word of the phrase, one by one, to see which words were correct or not. The recognition issues already we mentioned earlier made this process even more arduous.

Mondly is not for beginners, and may not be developed enough for those who just want to brush up on their other languages. It does not necessarily teach you a new language as much as it helps you learn a new language. The average instructor would help you learn words — the building blocks of most languages — before jumping into full sentences.

In the end, Learn Languages VR by Mondly will frustrate you, but if you stick through it, you’ll learn a thing or two. The app is free to download in the Oculus Store for the Samsung Gear VR.

Highs

  • Wide variety of languages to choose from at launch
  • Can detect voice even in noisy enviornments
  • Intuitive controls

Lows

  • Voice detection accuracy needs work
  • Doesn’t provide language fundamentals
Gaming

Minecraft Earth hands-on preview: The next big AR craze Is coming

Minecraft Earth is a new mobile augmented reality game from Microsoft that has Pokémon Go dead in its sites. The cute monster collect-them-all started the genre, but Minecraft Earth hopes to redefine it.
Mobile

Flex your thumbs (and your brain) with these fun texting games

Gaming consoles keep getting more advanced, but you can still have fun with the good old Latin alphabet. Here are our picks for the best texting games, so you can make the most fun out of that limited data plan or basic cell phone.
Smart Home

Make Alexa your own: How to change the digital assistant's voice

If you live in a country where English is not the primary language, or you're learning a second language and want to get all the practice you can get, you can set Alexa's default language to one of nine other languages.
Smart Home

Did you just get a brand-new Amazon Echo? Here's how to set it up

This hands-free, voice-controlled device can do so much. Here's how to set up your Amazon Echo, how to customize settings to get the experience you want, and how to set up more in-depth commands like smart device control.
Mobile

Get a grip on the lingo wherever you are with the Google Translate app

The ability to translate written text or the spoken word with the phone in your pocket is incredibly useful, and it’s completely free. Read our guide on how to use the Google Translate app for conversations, image translations, and more.
Mobile

Google’s new $999 augmented reality smartglasses are ready for business

Google unveiled Google Glass Enterprise Edition 2, a new version of its business-focused augmented reality wearable. The company's smartglasses ship with a faster processor, an updated camera, and safety frames from Smith Optics.
Product Review

Oculus Quest is the affordable VR rig we’ve been waiting for

Oculus announced that its Project Santa Cruz virtual reality headset will ship next year as the Oculus Quest, and we got to try out several new game titles on the Quest. Find out our impressions of VR without wires.
Cars

Volvo wants to use augmented reality tech to help design future cars

Volvo and Finnish tech firm Varjo developed an augmented reality headset that can be used while driving a real car. Volvo claims this will help speed up the development process of future cars.
News

Apple unveils ARKit 3 for more immersive augmented reality experiences

Apple has unveiled ARKit 3, which brings major improvements to its augmented reality platform. New features include people occlusion and motion capture, in addition to RealityKit and Reality Composer.
Computing

HTC’s Vive Pro Eye, a $1,600 VR headset with eye-tracking, is all business

HTC has finally released its Vive Pro Eye virtual reality headset in the United States, but before you get excited, it's important to note that it's not really for gamers. This headset is meant for commercial and business use.
Virtual Reality

Augmented reality brings VR to the real world in all sorts of exciting ways

Heard of virtual reality, but still wondering what augmented reality is? It's not too far a leap from the former -- it just blends the real world with the digital one in often unique and exciting ways.
Deals

Walmart offers the Oculus Go virtual reality headset with a $30 gift card

Are you in the market for a virtual reality headset? Walmart is running a great deal on the Oculus Go VR headset. Here's everything you need to know about this great standalone headset.
Gaming

Immerse yourself in a new universe with these incredible PSVR games

The PSVR surpassed expectations and it comes with an incredible catalog of games. There's plenty of amazing experiences to be had so we've put together a list of the best PSVR games available today.
Virtual Reality

Getting into VR is spendy. Which headset is truly worth your hard-earned cash?

Virtual reality has finally gone mainstream, but how do you find the best VR headset for you? Check out a few of our favorites, whether you want the best of the best or a budget alternative for your mobile device.