On a videogame mission to learn Arabic
To teach soldiers basic Arabic quickly, University of Southern California computer scientists are developing a system that uses artificial intelligence and computer game techniques.
The exams in the Rapid Tactical Language Training System created by the USC Viterbi School of Engineering’s Center for Research in Technology for Education (CARTE) and partners are videogame missions into AI-animated Arabic speaking virtual environments where soldier students have to successfully phrase questions and understand answers to pass.”Most adults,” said CARTE director W. Lewis Johnson, “find it extremely difficult to acquire even a rudimentary knowledge of a language, particularly in a short time.”
“We’re trying,” he continued, “to build an improved model of instruction, one that can be closely tailored to both the needs and the abilities of each individual student.”
An early version of the system was tried out in October 2003, by cadets at the U.S. Military Academy at West Point studying Arabic, who offered suggestions. December trials at Ft. Bragg by enlisted personnel were encouraging and led to further guidance on making the material accessible.
Johnson leads a six-person CARTE team that is spearheading the effort. DARPA and the Office of Naval Research are funding the work. The Rapid Tactical Language Training System is one of a number of instructional programs being developed as part of a initiative called DARPA Universal, Persistant, On-Demand Training War, or “DARWARS,” aimed at developing a heterogenous set of active learning tools.
Part of the system, the “Mission Skill Builder,” is somewhat like an intensive version of the language laboratory programs that have been in use for generations, in which students are exposed to words and phrases pronounced by native speakers, which they imitate, and practice using in sample dialogues.
Students learn the basics in instructional segments offered in the Mission Skill Builder interface shown above. Their understanding is then tested in the Mission Practice Environment.
“While it’s similar to drill-and-practice language programs that have been in use for some time, the Skill Builder incorporates some important innovations,” Johnson elaborated.
These include:
Speech recognition technology tailored for language learner speech, that is able to evaluate learner speech and detect common errors;
Pedagogical agent technology that provides the learner with tailored feedback on their performance; and
A learner model that dynamically keeps track of what aspects of the language the learner has mastered and what areas the learner is deficient in.
Along with linguistic skills, this section of the program also instructs students in non-linguistic cultural matters of importance in communication. “People don’t just communicate with words,” said Johnson.
“In face-to-face conversation, nonverbal behavior such as gesture, posture, gaze, head movements and facial expression play an important role in coordinating a successful exchange,” explained ISI research scientist Hannes Hogni Vilhjalmsson, a specialist in modeling human non-verbal communication. “Wrong interpretation of nonverbal cues or the wrong nonverbal responses can lead to serious misunderstanding and escalate hostility. Since these cues can depend on culture, it is important to include all of these behaviors when teaching conversation skills in a foreign language. ”
Vilhjalmsson says that “by exposing learners to realistic face-to-face situations and by training them to be culturally sensitive, we prepare them to become effective social players as well as speakers in the new language.”
Points covered in culture training include social skills necessary to build rapport with local people: appropriate degrees of politeness to use in different social situations, how to disagree with someone without offending, and how to respond to offers to hospitality.
Gesture training includes common Arabic gestures that a Westerner might misinterpret (for example, Arabs may roll their eyes to mean “no”) and American gestures (such as thumbs-up) that an Arab might misinterpret.
The examination or application part of the training system, the “Mission Practice Environment,” is still more innovative. It is designed to give students an unscripted, unpredictable, and therefore challenging test of their mastery of these elements.
In this segment, earphone and microphone wearing students control a uniformed figure moving through a videogame-like Lebanese village, complete with outdoor coffee bar. They meet Artificial-Intelligence-animated Arabic speakers, who (thanks to sophisticated voice recognition programs) can carry on free-form conversations.
The Mission Practice Envrionment offers a test of ability to carry on two-way communication in an appropriate maneer. The student controls the figure at the center, and speaks for the character in Arabic, using a microphone. The other characters, animated by AI, can understand what is said (if it is understandable Arabic) and respond. In the environment, gestures as well as words are significant.
“These AI figures can understand what the students say, if it’s said correctly – or won’t, if it isn’t. And they will respond appropriately,” said Johnson.
In the exercise, after exchanging greetings the student finds out the names of locals, the name of the place, the identity of the local headman and the location of his house, and must follow these directions through the game interface to get there.
“In typical videogame fashion, the idea is to get to the next level,” said Johnson. “In this game, in order to get to the next level, the learner has to master the linguistic skills.”
The program already has features designed to adapt it to each individual user, noting consistent errors or difficulties, which can be targeted for extensive or remedial practice.
So far, approximately 7 hours have been completed. The full program is anticipated to include about 80 hours of instruction, and introduce perhaps 500 carefully chosen words of the “Levantine” Arabic spoken in Lebanon to learners. If all goes as planned, the system may be deployed next year.
“We here in the Department of Foreign Languages are very excited about the Tactical Language Training System and the new capabilities that it can provide to military language learners, including our cadets,” said Colonel Stephen LaRocca of the Department of Foreign Languages at West Point.
“This system allows learners to rehearse real-world tasks in the most realistic environment technology can provide,” LaRocca continued. “It will be available wherever and whenever the learner is available, and activities can be repeated as often as the learner desires. … It has the potential to greatly expand speaking opportunities in a meaningful, motivating context.”
Working with CARTE on the project are USC’s Integrated Media Systems Center; UCLA’s Center for Research on Evaluation, Standards, and Student Testing; and the Micro Analysis & Design Inc. firm of Boulder, CO.
CARTE is headquartered at USC Engineering’s Information Sciences Institute. Besides Johnson and Vilhjalmsson, the ISI researchers involved include Stacy Marsella, Catherine M. LaBore, Carole Beal, Nicolaus Mote, Shumin Wu, Hartmut Neven, Ulf Hermjakob, and Mei Si.
Published on January 9th, 2004
Last updated on August 9th, 2021
