AI emotional reasoning through audios and videos (Credit: Midjourney)
Have you ever interacted with an AI-powered customer support agent only to find it strangely nonempathetic, cold, disconnected and unable to understand how you’re feeling?
Unlike humans, AI does not actually experience emotions. Instead, it can appear to understand emotions by using logic to infer them from audio and visual cues — a capability known as emotion reasoning.
This ability is often found in multimodal large language models (MLLMs), advanced AI systems that can understand and analyze multiple types of data, like images, audio, video and text, simultaneously.
However, today’s AI technologies still face significant limitations in emotion reasoning, often hallucinating or completely mislabeling how people feel. These errors can compromise the reliability of AI agents in social interactions and other complex emotion reasoning scenarios.
USC researcher Mohammad Soleymani led a study investigating why AI models fail to associate audiovisual cues with the correct emotions and how to “debug AI’s logic” by proposing a new optimization framework that helps ensure emotional judgments are logically grounded in actual evidence.
The project, which began last spring, resulted in the paper, “AVERE: Improving Audiovisual Emotion Reasoning With Preference Optimization,” which was accepted to the International Conference on Learning Representations (ICLR) 2026.
The study was funded by the Army Research Office (ARO), the National Science Foundation (NSF) and the National Institute of Mental Health (NIMH).
The research team also included Soleymani’s PhD students Jiacheng Pang and Ashutosh Chaubey, as well as Maksim Siniukov.
Soleymani is a research associate professor at USC Viterbi School of Engineering and the USC Mark and Mary Stevens School of Computing and AI‘s Thomas Lord Department of Computer Science, he is also a research scientist at the USC Institute for Creative Technologies.
How Does AI Think About Emotions: Designing an EQ Test for AI
“Our first step was to understand and quantify these challenges and pinpoint where and how exactly these multimodal models fail in the emotion reasoning process,” explained Soleymani.
The team extracted video clips and created a benchmark called EmoReAlM that consists of 4,000 multiple-choice questions.
AI models are given videos along with separate audio and visual captions and asked questions to correctly identify the emotions present in the clips and the evidence behind their answers.
The researchers emphasized that this benchmark was a crucial step forward because previous studies had not systematically evaluated the failure modes of MLLMs.
AI Selectively Ignores What It Sees and Hears
Soleymani’s team identified two categories of AI mistakes, or failure modes: reasoning errors and perception errors.
They found that reasoning errors occur when a model correctly identifies an emotion but justifies it using irrelevant audiovisual cues. Those cues are actually present in the video clips — they are just completely unrelated to the emotions. For example, MLLMs may say, “This man is scared because he’s bald,” or “The girl is happy because her sweater is green.”
Perception errors, on the other hand, involve audio or visual evidence that does not exist in the videos and is instead completely made up, or hallucinated, by the models. For example, the researchers found models making claims that a person is sad because crying sounds were detected in a clip, when in reality the audio is completely silent.
But why does AI do that?
The team observed that the models were taking a cognitive shortcut.
If a text prompt mentions or asks about a specific emotion, such as “sadness,” AI’s “brain” hyper focuses on related concepts like “crying” and completely ignores what actually happens in the video or audio.
Models begin to generate a chain of reasoning that satisfies their own initial prediction instead of looking for real, grounded evidence from the clip. Instead, AI fabricates evidence, hallucinating wiped tears or the sound of crying to fit its own “self-fulfilling” beliefs.
The team also attributed the failure modes to AI’s “language bias,” meaning MLLMs still treat text as a higher priority than audio and visual cues when reasoning and making judgments. The study also found that models performed poorly on “disgust,” which the researchers attributed to the inherent ambiguity of the emotion and a lack of sufficient training data for that specific category.
The MLLMs also struggled to manage conflicting signals, such as when a caption indicates “crying” but the person in the video is crying tears of joy.
Teaching AI to Reason With Real Evidence
To get rid of AI’s failure modes and language biases, Soleymani team proposed a two-part solution.
The first uses “AVEm-DPO (Audiovisual Emotion Direct Preference Optimization).” to create preference pairs where the model is shown a good, logical and grounded output versus a bad, hallucinated or misattributed output, forcing it to prioritize the correct logic.
After applying their AVEm-DPO solution, the researchers used “attention maps” to show that the model had actually shifted its focus, as it could pay significantly more attention to audio and video information rather than just the text prompt.
Next, by identifying these shortcuts, the team was able to develop Text Prior Debiasing (TPD), which specifically penalizes AI for providing answers that could be guessed through text alone, forcing the model to “ground” its reasoning in actual audiovisual evidence. This effectively tells the AI that a response generated purely from the prompt is a “second-class citizen” compared with one that uses audio and visual evidence.
Published on September 14th, 2026
Last updated on September 16th, 2026