Whether it’s apps like Duolingo or the ease of travel, there are plenty of ways technology has made it more straightforward to learn a second (or third or fourth …) language. Now, IBM Research and New York’s Rensselaer Polytechnic Institute have come up with an entirely new high-tech approach — and it totally reminds us of the Vulcan school from 2009’s Star Trek movie.
Called the Cognitive Immersive Room (CIR), it pairs an A.I.-powered chatbot smart assistant with a 360-degree panoramic display system to place users into a variety of immersive locations to try out their language skills. Currently, it’s being used for Mandarin, which is widely considered to be among the more difficult languages for Westerners to learn. The CIR setup drops students into scenarios like a restaurant in China and a tai chi class, where they can put their Mandarin to the test.
“The Cognitive Immersive Classroom is a very important use case for us,” Hui Su, Director of the Cognitive and Immersive Systems Lab at IBM Research, told Digital Trends. “We are currently focusing on language learning, and are building the classroom for students who study Mandarin as a second language. In this classroom, students are immersed into a 360-degree [environment], surrounded by real-life scenes such as a restaurant, street market, and garden. They can talk to the avatars, such as the waiters and salesmen on the screens, and perform tasks like ordering drinks and food and buying products. These tasks are developed as games for students to complete. In these games, the students will be able to practice and learn Chinese languages in a culture-rich environment, and talk to the A.I. agents who can understand what they just said.”
To make the experience even more immersive, the room is kitted out with cameras, Kinect devices, and microphones. This makes it possible to, for instance, point at an object and say “what is that?” and have the question answered. The microphones, meanwhile, can pick up on every nuance of a speaker’s words.
“A pitch contour analysis function was developed to visualize the difference between the students’ pronunciations and those by native speakers, so that students could see easily where their tone pronunciations need to be improved,” Su said. “The classroom is a brand-new initiative to integrate A.I. technologies and human-scale immersive technologies together for Mandarin teaching. We are using IBM Watson speech recognition and natural language understanding for English and Chinese.”
Coming soon to a classroom near you. Or, at least, we really, really hope it is.
