Skip to main content

Learn Languages VR: Our first take

Bonjour, mon ami! Mondly uses VR to teach new languages

What if you could have a conversation with someone speaking in a different language, without being spoon fed individual words, whenever and wherever you want? This has long been the holy grail of language, and software has chased it for decades — with marginal success.

Yet there’s new hope to be found in virtual reaility, and Learn Languages VR by Mondly, the company’s first virtual reality application, shows why. In Mondly VR, you don’t interact with an instructor. Instead, you have conversations with a chat bot represented by a variety of digital characters, which asks you questions in the language you have selected.

During your conversations you are prompted with common responses to what the character says, and every answer you give is transcribed on screen, so you can see where you messed up. If you answer correctly, a green check mark will hover over the transcription of what you just said. Answer wrong, and the character will repeat the question. Not many human teachers would have this level of patience, but the chatbot is ready to repeat as often as needed.

Learn Languages VR doesn’t stop at chatbots. People learn best through real-world practice, and the application emulates that. You can choose from three VR experiences that test you knowledge — ordering at a restaurant, meeting someone on a train, and checking into a hotel.

So, can you really learn a new language in virtual reality? We gave it a go.

Robot Learning

Mondly is based on a server-side chat bot, which enables the app to speak with you without an actual person feeding it answers. Popular language learning program Rosetta Stone only teaches you new languages in conversations if you schedule a video chat session with one of its tutors. Being able to hear a voice recite a question often times made it easier to understand the pronunciation and context of how to use suggested phrases. And if you are planning to vacation in Barcelona, knowing how to say “can I get a cab” rather than simply a “cab” will certainly help you fit in.

The Mondly experience leverages the immersive quality of VR by the amount of interactivity you can access with just voice and head movements. You stare at the character in your simulation and as you speak, which is a much more natural environment than looking at a book or screen. With just the slightest head movement, you can view and access a transcription of what the chatbot said, see answer suggestions, view English translations, start audio recitations in the selected language, and more.

To get an English translations of suggestions and the chatbot’s answer, you simply hover the cursor over the phrase, and the translation will appear below. Hovering over the speaker icon near any phrase will prompt a voice to recite it back in the selected language. This helps pick up the nuances of languages predicated on vocal inflections, such as Japanese. But, sometimes the app accepts an answer that correctly pronounces a part of the phrase with a different inflection, if the app feels you got the gist of the response across.

Can You Hear Me Now?

The most important part of Mondly is not how intelligent it is, but how well it can hear. Learn Languages VR relies on the illusion of conversing with a person, and that means constant coordination between audio and visual components in order to make you feel like you are somewhere you actually are not. If you are learning a new language in VR, any noticeable lag or miscommunication between you and the instructor means the illusion is gone, which destroys the experience.

“The toughest challenge was to make the speech interaction feel natural in VR.”

“The toughest challenge was to make the speech interaction feel natural in VR,” Mondly co-founder Alexandru Iliescu wrote in a press release for the app. The primacy of sound to the experience is paramount. Before each experience you are prompted to remain quiet as Mondly’s voice detection system calibrates the microphone to the room’s background noise in order to ignore unintended sounds. I used Mondly in a silent office space and in a communal area with constant chatter around, and noticed no difference in the voice recognition.

Even so, the software has some issues to work out. Mondly’s voice recognition is consistently perfect…as long as it’s asked to handle phrases just a few words long. While trying to ask for a beer at a virtual restaurant in French, I could see only parts of my response transcribed.

Seeing the app wasn’t working, We decided to slowly say the sentence word-by-word to see if I was jumbling the pronunciation. When we did, the app displayed different words than what I was speaking, consistently. I yelled “Bier” in French 4 times, and was met “Heir.” Eventually, we gave up.

Too Smart To Teach

Mondly teaching you how to speak a different language through conversations is a double-edged sword. It is a crash course on context, verb tenses, and pronunciation in a language you probably never spoke before. The learning curve can be steep, and the app tends to throw users into the deep end before teaching them to swim.

We found a lesson might start with a simple phrase, then follow up by asking us to recite an eight word sentence introducing ourselves. This seems to make the app most useful to those who can read the language they wish to learn, but don’t know how to form sentences. Those Spanish classes you took in high school could come in handy.

Mondly VR

Each experience yields different levels of difficulty, as well. We talked through a light conversation on the train in German without much trouble. Trying to reserve a room with a shower was considerably more complex. After numerous failed attempts at complete phrases, we would repeat each word of the phrase, one by one, to see which words were correct or not. The recognition issues already we mentioned earlier made this process even more arduous.

Mondly is not for beginners, and may not be developed enough for those who just want to brush up on their other languages. It does not necessarily teach you a new language as much as it helps you learn a new language. The average instructor would help you learn words — the building blocks of most languages — before jumping into full sentences.

In the end, Learn Languages VR by Mondly will frustrate you, but if you stick through it, you’ll learn a thing or two. The app is free to download in the Oculus Store for the Samsung Gear VR.


  • Wide variety of languages to choose from at launch
  • Can detect voice even in noisy enviornments
  • Intuitive controls


  • Voice detection accuracy needs work
  • Doesn’t provide language fundamentals

Editors' Recommendations

Keith Nelson Jr.
Former Digital Trends Contributor
Keith Nelson Jr is a music/tech journalist making big pictures by connecting dots. Born and raised in Brooklyn, NY he…
Apple’s XR headset could get one of the Mac’s best features
Apple VR Headset Concept by Antonio De Rosa

Apple is known for the strong ecosystem that lets all its devices work pretty seamlessly together. A new patent suggests the company’s upcoming Reality Pro headset will be a full-fledged member of this ecosystem -- and get one of Apple’s best features in the process.

According to the patent, Apple might bring its Continuity system to its upcoming mixed-reality headset. That means you’ll be able to send work from one device to another with just your eyes, all through the power of the headset’s augmented reality tech.

Read more
Apple’s second-generation VR headset is already in the works
A rendering of an Apple mixed-reality headset (Reality Pro) in a gold color seen from the front.

Apple’s first mixed-reality headset hasn’t even been released, yet the company is already working on a second-generation model that will bring even more power and performance to the table, according to a new report. That suggests we might not have long to wait before the successor device hits the shelves.

The news comes from Bloomberg reporter Mark Gurman in his Power On newsletter. Gurman claims that the first edition of the headset -- likely dubbed Reality Pro -- will come with the same M2 chip that powers the latest MacBook Air, plus a separate chip that will handle augmented reality (AR) and virtual reality (VR) experiences.

Read more
We now know why Apple’s Reality Pro headset was delayed
A rendering of four Apple mixed-reality headsets (Reality Pro) in various colors sitting on a surface.

Until last week, Apple’s Reality Pro mixed-reality headset was expected at a spring event. Yet that news was spoiled when Bloomberg journalist Mark Gurman reported that the device had been pushed back to June, and he has now shared exactly why that is -- along with another surprising revelation.

Why June, you ask? Well, that’s when Apple hosts its Worldwide Developers Conference (WWDC), a massive showcase event where developers can learn about the latest Apple software releases and get hands-on in a weeklong series of demonstrations and seminars.

Read more