What if you could have a conversation with someone speaking in a different language, without being spoon fed individual words, whenever and wherever you want? This has long been the holy grail of language, and software has chased it for decades — with marginal success.
Yet there’s new hope to be found in virtual reaility, and Learn Languages VR by Mondly, the company’s first virtual reality application, shows why. In Mondly VR, you don’t interact with an instructor. Instead, you have conversations with a chat bot represented by a variety of digital characters, which asks you questions in the language you have selected.
During your conversations you are prompted with common responses to what the character says, and every answer you give is transcribed on screen, so you can see where you messed up. If you answer correctly, a green check mark will hover over the transcription of what you just said. Answer wrong, and the character will repeat the question. Not many human teachers would have this level of patience, but the chatbot is ready to repeat as often as needed.
Learn Languages VR doesn’t stop at chatbots. People learn best through real-world practice, and the application emulates that. You can choose from three VR experiences that test you knowledge — ordering at a restaurant, meeting someone on a train, and checking into a hotel.
So, can you really learn a new language in virtual reality? We gave it a go.
Robot Learning
Mondly is based on a server-side chat bot, which enables the app to speak with you without an actual person feeding it answers. Popular language learning program Rosetta Stone only teaches you new languages in conversations if you schedule a video chat session with one of its tutors. Being able to hear a voice recite a question often times made it easier to understand the pronunciation and context of how to use suggested phrases. And if you are planning to vacation in Barcelona, knowing how to say “can I get a cab” rather than simply a “cab” will certainly help you fit in.
The Mondly experience leverages the immersive quality of VR by the amount of interactivity you can access with just voice and head movements. You stare at the character in your simulation and as you speak, which is a much more natural environment than looking at a book or screen. With just the slightest head movement, you can view and access a transcription of what the chatbot said, see answer suggestions, view English translations, start audio recitations in the selected language, and more.
To get an English translations of suggestions and the chatbot’s answer, you simply hover the cursor over the phrase, and the translation will appear below. Hovering over the speaker icon near any phrase will prompt a voice to recite it back in the selected language. This helps pick up the nuances of languages predicated on vocal inflections, such as Japanese. But, sometimes the app accepts an answer that correctly pronounces a part of the phrase with a different inflection, if the app feels you got the gist of the response across.
Can You Hear Me Now?
The most important part of Mondly is not how intelligent it is, but how well it can hear. Learn Languages VR relies on the illusion of conversing with a person, and that means constant coordination between audio and visual components in order to make you feel like you are somewhere you actually are not. If you are learning a new language in VR, any noticeable lag or miscommunication between you and the instructor means the illusion is gone, which destroys the experience.
“The toughest challenge was to make the speech interaction feel natural in VR.”
“The toughest challenge was to make the speech interaction feel natural in VR,” Mondly co-founder Alexandru Iliescu wrote in a press release for the app. The primacy of sound to the experience is paramount. Before each experience you are prompted to remain quiet as Mondly’s voice detection system calibrates the microphone to the room’s background noise in order to ignore unintended sounds. I used Mondly in a silent office space and in a communal area with constant chatter around, and noticed no difference in the voice recognition.
Even so, the software has some issues to work out. Mondly’s voice recognition is consistently perfect…as long as it’s asked to handle phrases just a few words long. While trying to ask for a beer at a virtual restaurant in French, I could see only parts of my response transcribed.
Seeing the app wasn’t working, We decided to slowly say the sentence word-by-word to see if I was jumbling the pronunciation. When we did, the app displayed different words than what I was speaking, consistently. I yelled “Bier” in French 4 times, and was met “Heir.” Eventually, we gave up.
Too Smart To Teach
Mondly teaching you how to speak a different language through conversations is a double-edged sword. It is a crash course on context, verb tenses, and pronunciation in a language you probably never spoke before. The learning curve can be steep, and the app tends to throw users into the deep end before teaching them to swim.
We found a lesson might start with a simple phrase, then follow up by asking us to recite an eight word sentence introducing ourselves. This seems to make the app most useful to those who can read the language they wish to learn, but don’t know how to form sentences. Those Spanish classes you took in high school could come in handy.
Each experience yields different levels of difficulty, as well. We talked through a light conversation on the train in German without much trouble. Trying to reserve a room with a shower was considerably more complex. After numerous failed attempts at complete phrases, we would repeat each word of the phrase, one by one, to see which words were correct or not. The recognition issues already we mentioned earlier made this process even more arduous.
Mondly is not for beginners, and may not be developed enough for those who just want to brush up on their other languages. It does not necessarily teach you a new language as much as it helps you learn a new language. The average instructor would help you learn words — the building blocks of most languages — before jumping into full sentences.
In the end, Learn Languages VR by Mondly will frustrate you, but if you stick through it, you’ll learn a thing or two. The app is free to download in the Oculus Store for the Samsung Gear VR.
Highs
- Wide variety of languages to choose from at launch
- Can detect voice even in noisy enviornments
- Intuitive controls
Lows
- Voice detection accuracy needs work
- Doesn’t provide language fundamentals
