It’s Wednesday night and I’m at my kitchen table staring into my laptop as I pour all the bile I can muster into three little words: “I love you.”
My neighbors might assume I’m engaged in a melodramatic phone call to an ex-partner, or perhaps some sort of acting exercise, but I’m actually testing the limits of a new demo from HumeA Manhattan-based startup claims to have developed “the world’s first voice AI with emotional intelligence.”
“We’re training a large language model that also understands your tone of voice,” says Hume’s CEO and Chief Scientist Alan Cowen. “What it enables … is to be able to predict how a given speech utterance or sentence will elicit patterns of emotion.”
In other words, Hume claims to recognize the emotion in our voices (and in another, non-public version, facial expressions) and respond empathically.
Bolstered by Open AI’s introduction of the new, more “emotional” GPT4o this Mayso-called emotional AI is increasingly big business. Hume raised $50 million in its second round of funding in March, and the industry’s value was predicted to reach more than $50 billion this year. But Prof Andrew McStay, director of the Emotional AI Lab at Bangor University, suggests that such predictions are meaningless. “Emotion is such a fundamental dimension of human life that if you can understand, measure and respond to emotion in natural ways, it has implications far beyond $50 billion,” he says.
Possible applications range from better video games and less frustrating helplines to Orwell-worthy surveillance and mass emotional manipulation. But is it really possible for AI to accurately read our emotions, and if some form of this technology is on the way, regardless, how should we deal with it?
“I appreciate your kind words, I’m here to support you,” Hume’s Empathic Voice Interface (EVI) replies in a friendly, almost-human voice as my declaration of love transcribed and analyzed appears on the screen: 1 (of 1 ) for “love”, 0.642 for “adoration”, and 0.601 for “romance”.
While the failure to detect any negative feeling may be due to bad acting on my part, I get the impression that more weight is being given to my words than to my tone of voice, and when I take this up to Cowen, he tells me it is difficult for the model to understand situations it has not encountered before. “It understands your tone of voice,” he says. “But I don’t think anyone has ever heard that ‘I love you’ in that tone of voice.”
Maybe not, but should a truly empathetic AI recognize that humans rarely wear their hearts on their sleeves? As Robert De Niro, a master of portraying human emotions, once remarked: “People don’t try to show their feelings, they try to hide them.”
Cowen says Hume’s goal is just to understand people’s overt expressions, and in fairness the EVI is remarkably responsive and naturalistic when approached sincerely – but what would an AI do with our less simple behaviour?
EEarlier this year, associate professor Matt Coler and his team at the University of Groningen’s speech technology lab used data from American sitcoms including Friends and The big bang theory on train an AI that can recognize sarcasm.
That sounds useful, you might think, and Coler argues it is. “As we watch machines permeate more and more human life,” he says, “it becomes our duty to make sure that those machines can actually help people in a useful way.”
Coler and his colleagues hope that the company’s work with sarcasm will lead to advances in other linguistic devices, including irony, exaggeration, and politeness, enabling more natural and accessible human-machine interactions, and they’re off to an impressive start. The model detects sarcasm accurately 75% of the time, but the remaining 25% raises questions such as: how much license should we give machines to interpret our intentions and feelings; and what degree of accuracy would that license require?
Emotional AI’s fundamental problem is that we cannot definitively say what emotions are. “Put a room of psychologists together and you’re going to have fundamental disagreements,” says McStay. “There is no baseline, agreed-upon definition of what emotion is.”
There is also no agreement on how emotions are expressed. Lisa Feldman Barrett is a professor of psychology at Northeastern University in Boston, Massachusetts, and in 2019 she and four other scientists came together with a simple question: can accurately infer our emotions from facial movements alone? “We read and summarized over 1,000 papers,” says Barrett. “And we did something that no one else had done so far: we came to a consensus about what the data was saying.”
The consensus? We can’t.
“This is very relevant to emotional AI,” says Barrett. “Because most companies I’m aware of still promise that you can look at a face and tell if someone’s angry or sad or scared or what have you. And that is clearly not the case.”
“An emotionally intelligent human don’t usually claim that they can accurately put a label on everything everyone is saying and tell you this person is currently feeling 80% angry, 18% scared and 2% sad,” says Edward B Kang, an assistant professor at the New York University who writes about the intersection of AI and sound. “In fact, that sounds to me like the opposite of what an emotionally intelligent person would say.”
Adding to this is the infamous problem of AI bias. “Your algorithms are only as good as the training material,” says Barrett. “And if your training material is biased in some way, then you entrench that bias in code.”
Research showed that some emotional AIs disproportionately attribute negative emotions to the faces of black people, which would have clear and worrying implications if deployed in areas such as recruitment, performance evaluations, medical diagnostics or policing. “We have to bring [AI bias] at the forefront of the conversation and design of new technologies,” says Randi Williams, program manager at the Algorithmic Justice League (AJL), an organization working to raise awareness of bias in AI.
So there are concerns about emotional AI not working as it should, but what if it’s working too well?
“When we have AI systems tapping into the most human part of ourselves, there is a high risk of individuals being manipulated for commercial or political gain,” says Williams, four years after a whistleblower’s documents were revealed have that the “industrial scale” on which Cambridge Analytica used Facebook data and psychological profiling to manipulate voters, emotional AI seems ripe for abuse.
As is customary in the AI industry, Hume has made appointments to a safety board – the Hume Initiative – that counts its CEO among its members. The initiative’s ethical guidelines describe itself as a “non-profit effort charting an ethical path for empathic AI”, and include an extensive list of “conditionally supported use cases” in fields such as arts and culture, communication, education and health, and a much smaller list of “unsupported use cases” citing broad categories such as manipulation and deception, with some examples including psychological warfare, deep spoofing and “optimizing for user engagement”.
“We only allow developers to deploy their applications if they are listed as supported use cases,” Cowen says via email. “Of course, the Hume Initiative welcomes feedback and is open to reviewing new use cases as they arise.”
As with all AI, designing security strategies that can keep up with the speed of development is a challenge.
Approved in May 2024, the European Union AI Law bans the use of AI to manipulate human behavior and bans emotion recognition technology from spaces including the workplace and schools, but draws a distinction between identifying expressions of emotion (which will be allowed), and inferring an individual’s emotional state from it (which wouldn’t). Under the law, a call center manager using emotional AI for monitoring can likely discipline an employee if the AI says they sound grumpy on calls, just as long as there is no distraction that they are in fact grumpy. “Frankly, anyone can still use that kind of technology without making an explicit inference about a person’s inner emotions and making decisions that might affect them,” says McStay.
The UK does not have specific legislation, but McStay’s work with the Emotional AI Lab has helped inform the policy position of the Information Commissioner’s Office, which in 2022 companies warned to avoid “emotional analysis” or incur fines, citing the field’s “pseudo-scientific” nature.
In part, proposals of pseudoscience come from the problem of trying to infer emotional truths from large data sets. “You can run a study where you find an average,” explains Lisa Feldman Barrett. “But if you went to any individual person in any individual study, they wouldn’t have that average.”
However, making predictions from statistical abstractions does not mean that an AI cannot be right, and certain uses of emotional AI may be able to circumvent some of these issues.
A week after putting Hume’s EVI through its paces, I have a decidedly more candid conversation with Lennart Högman, assistant professor of psychology at Stockholm University. Högman tells me about the joys of raising his two sons, then I describe a particularly good day from my childhood, and once we’ve shared these happy memories, he feeds the video of our Zoom call into software his team is developing have to analyze people’s emotions together. “We look at the interaction,” he says. “So it’s not one person showing something, it’s two people interacting in a specific context, like psychotherapy.”
Högman suggests the software, which relies in part on analyzing facial expressions, could be used to track a patient’s emotions over time, and would provide a useful tool for therapists whose services are increasingly delivered online by helping to determine the progress of treatment, identify persistent responses to certain topics, and monitor alignment between patient and therapist. “Alliance has been shown to be perhaps the most important factor in psychotherapy,” says Högman.
While the software analyzes our conversation frame by frame, Högman emphasizes that it is still in development, but the results are interesting. Scrolling through the video and accompanying graphs, we see moments where our emotions seem to align, where we mirror each other’s body language, and even when one of us seems to be more dominant in the conversation.
Insights like these can potentially grease the wheels of business, diplomacy and even creative thinking. Högman’s team has yet to conduct published research suggesting a correlation between emotional synchronization and successful collaboration on creative tasks. But there is inevitably room for abuse. “When both parties in a negotiation have access to AI analytics tools, the dynamics undoubtedly change,” explains Högman. “The benefits of AI may be negated as each side becomes more sophisticated in their strategies.”
As with any new technology, the impact of emotional AI will ultimately come down to the intentions of those who control it. As Randi Williams of the AJL explains, “To successfully embrace these systems as a society, we need to understand how users’ interests are misaligned with the institutions that create the technology.”
Until we’ve done that and acted on it, emotional AI is likely to evoke mixed feelings.
