Learn Languages VR: Our first take

Bonjour, mon ami! Mondly uses VR to teach new languages

What if you could have a conversation with someone speaking in a different language, without being spoon fed individual words, whenever and wherever you want? This has long been the holy grail of language, and software has chased it for decades — with marginal success.

Yet there’s new hope to be found in virtual reaility, and Learn Languages VR by Mondly, the company’s first virtual reality application, shows why. In Mondly VR, you don’t interact with an instructor. Instead, you have conversations with a chat bot represented by a variety of digital characters, which asks you questions in the language you have selected.

During your conversations you are prompted with common responses to what the character says, and every answer you give is transcribed on screen, so you can see where you messed up. If you answer correctly, a green check mark will hover over the transcription of what you just said. Answer wrong, and the character will repeat the question. Not many human teachers would have this level of patience, but the chatbot is ready to repeat as often as needed.

Learn Languages VR doesn’t stop at chatbots. People learn best through real-world practice, and the application emulates that. You can choose from three VR experiences that test you knowledge — ordering at a restaurant, meeting someone on a train, and checking into a hotel.

So, can you really learn a new language in virtual reality? We gave it a go.

Robot Learning

Mondly is based on a server-side chat bot, which enables the app to speak with you without an actual person feeding it answers. Popular language learning program Rosetta Stone only teaches you new languages in conversations if you schedule a video chat session with one of its tutors. Being able to hear a voice recite a question often times made it easier to understand the pronunciation and context of how to use suggested phrases. And if you are planning to vacation in Barcelona, knowing how to say “can I get a cab” rather than simply a “cab” will certainly help you fit in.

The Mondly experience leverages the immersive quality of VR by the amount of interactivity you can access with just voice and head movements. You stare at the character in your simulation and as you speak, which is a much more natural environment than looking at a book or screen. With just the slightest head movement, you can view and access a transcription of what the chatbot said, see answer suggestions, view English translations, start audio recitations in the selected language, and more.

To get an English translations of suggestions and the chatbot’s answer, you simply hover the cursor over the phrase, and the translation will appear below. Hovering over the speaker icon near any phrase will prompt a voice to recite it back in the selected language. This helps pick up the nuances of languages predicated on vocal inflections, such as Japanese. But, sometimes the app accepts an answer that correctly pronounces a part of the phrase with a different inflection, if the app feels you got the gist of the response across.

Can You Hear Me Now?

The most important part of Mondly is not how intelligent it is, but how well it can hear. Learn Languages VR relies on the illusion of conversing with a person, and that means constant coordination between audio and visual components in order to make you feel like you are somewhere you actually are not. If you are learning a new language in VR, any noticeable lag or miscommunication between you and the instructor means the illusion is gone, which destroys the experience.

“The toughest challenge was to make the speech interaction feel natural in VR.”

“The toughest challenge was to make the speech interaction feel natural in VR,” Mondly co-founder Alexandru Iliescu wrote in a press release for the app. The primacy of sound to the experience is paramount. Before each experience you are prompted to remain quiet as Mondly’s voice detection system calibrates the microphone to the room’s background noise in order to ignore unintended sounds. I used Mondly in a silent office space and in a communal area with constant chatter around, and noticed no difference in the voice recognition.

Even so, the software has some issues to work out. Mondly’s voice recognition is consistently perfect…as long as it’s asked to handle phrases just a few words long. While trying to ask for a beer at a virtual restaurant in French, I could see only parts of my response transcribed.

Seeing the app wasn’t working, We decided to slowly say the sentence word-by-word to see if I was jumbling the pronunciation. When we did, the app displayed different words than what I was speaking, consistently. I yelled “Bier” in French 4 times, and was met “Heir.” Eventually, we gave up.

Too Smart To Teach

Mondly teaching you how to speak a different language through conversations is a double-edged sword. It is a crash course on context, verb tenses, and pronunciation in a language you probably never spoke before. The learning curve can be steep, and the app tends to throw users into the deep end before teaching them to swim.

We found a lesson might start with a simple phrase, then follow up by asking us to recite an eight word sentence introducing ourselves. This seems to make the app most useful to those who can read the language they wish to learn, but don’t know how to form sentences. Those Spanish classes you took in high school could come in handy.

Mondly VR

Each experience yields different levels of difficulty, as well. We talked through a light conversation on the train in German without much trouble. Trying to reserve a room with a shower was considerably more complex. After numerous failed attempts at complete phrases, we would repeat each word of the phrase, one by one, to see which words were correct or not. The recognition issues already we mentioned earlier made this process even more arduous.

Mondly is not for beginners, and may not be developed enough for those who just want to brush up on their other languages. It does not necessarily teach you a new language as much as it helps you learn a new language. The average instructor would help you learn words — the building blocks of most languages — before jumping into full sentences.

In the end, Learn Languages VR by Mondly will frustrate you, but if you stick through it, you’ll learn a thing or two. The app is free to download in the Oculus Store for the Samsung Gear VR.

Highs

  • Wide variety of languages to choose from at launch
  • Can detect voice even in noisy enviornments
  • Intuitive controls

Lows

  • Voice detection accuracy needs work
  • Doesn’t provide language fundamentals
Emerging Tech

What’s it like to die? This VR experience puts doctors in a dying man’s shoes

A new VR experience depicts the last days of a lung cancer patient. It is already being used as a teaching tool in hospices and medical schools. But can VR really make us more empathetic?
Virtual Reality

Think virtual reality is just for games? These awesome apps will change your mind

Virtual reality isn't all about gaming. Swim with turtles, paint in 3D, and immerse yourself in some unique experiences the platform has to offer with our curated list of the best VR apps.
Product Review

Mediocre battery and a big notch slight Google's otherwise perfect Pixel phone

Google’s Pixel 3 XL has two big flaws: The gigantic notch on the front, and mediocre battery life. That being said, this is the best Android experience you can find in a smartphone today.
Gaming

The best HTC Vive games available today

So you’re considering an HTC Vive, but don't know which games to get? Our list of 25 of the best HTC Vive games will help you out, whether you're into rhythm-based gaming, interstellar dogfights, or something else entirely.
Mobile

These 100 best iPhone apps will turn your phone into a jack-of-all-trades

The iPhone is the most popular smartphone in the world, and we want to bring out the best in yours. Behold our comprehensive list of the best iPhone apps, from time-saving productivity tools to fun apps you won’t be able to put down.
Photography

Intel’s augmented reality brings Red Bull Rampage into your living room

The extreme mountain biking action of Red Bull Rampage is coming to your living room thanks to the power of augmented reality and 360-degree video that lets fans load a scale replica of the course right in their phones.
Mobile

From the road to your wrist, see how Android has evolved over the past 10 years

Android started out as just a mobile operating system, but 10 years in it's pretty much everywhere. Check out our round-up of all the different Android variations that have cropped up so far, and what might be coming in the future.
Photography

With flip-out lenses, the Vuze XR transforms from 2D to 180-degree VR camera

The Vuze XR is a compact, dual-lens camera with an integrated handle and a neat party trick: The back-to-back lenses can flip forward to transition from 360-degree, two-dimensional video to 180-degree, three-dimensional video.
Computing

Following the Portal, augmented reality glasses may be Facebook’s next step

Following the launch of its Portal smart display, Facebook says it is working on AR glasses, possibly in a move at challenging both Apple and Google and perhaps to rise up in the hardware scene. 
Emerging Tech

Here’s all the best gear and gadgetry you can snag for $100 or less

A $100 bill can get you further than you might think -- so long as you know where to look. Check out our picks for the best tech under $100, whether you're in the market for headphones or a virtual-reality headset.
Computing

Oculus VR could upgrade the Rift with a new display in 2019

Oculus could be set to release a new version of its Rift headset in 2019, but it will be more of a modest upgrade than a true sequel. The Rift S, as its purportedly called, will have a new display, and inside-out tracking.
Mobile

Google awarded patent for using eye tracking to detect expressions in VR

Google was awarded a patent that involves using eye tracking to infer facial expressions using machine learning in virtual reality. The tech could help make virtual reality a whole lot more immersive than it already is.
Deals

Cyber Monday 2018: When it takes place and where to find the best deals

Cyber Monday is still a ways off, but it's never too early to start planning ahead. With so many different deals to choose from during one of the biggest shopping holidays of the year, going in with a little know-how makes all the…
Gaming

Immerse yourself in a new universe with these incredible PSVR games

The PSVR has surpassed expectations and along with it comes an incredible catalog of games. There's plenty of amazing experiences to be had so we've put together a list of the best PSVR games available today.