“Wa moment, wait a moment. You haven’t heard anything yet.” So went the first line of audible dialogue in a feature film, 1927s The Jazz Singer. It was one of the first times that mass media transmitted the image and sound of a scene together, and the audience was mesmerized.
There have been improvements since then: black and white has become color, frame rates and resolutions have increased, and sound quality has improved, but the media we consume still overwhelmingly, if not exclusively, caters to our eyes and ears.
With the average person’s screen time now nearly seven hours a day, and much of that time spent indoors, our overreliance on sight and sound has only intensified. But given that humans are animals with five (or probably many more) senses, do we neglect our other faculties, and what does this do to us?
Many psychologists categorize our main senses as either rational or emotional, and there is evidence to support this. “Smell [and taste are] directly linked to the emotional processing areas of the brain,” says Charles Spence, a professor of experimental psychology at the University of Oxford, “while the rational senses such as hearing and vision are processed in the cortex.” In fact, says Spence, more than half of the neocortex – itself more than half the volume of the brain – is given over to processing what we see.
There is no denying that we are highly visual creatures and this is partly why our media is primarily audiovisual. “I think it’s mostly driven by the fact that much of the information we consider important today can be conveyed via visual or auditory means,” says Meike Scheller, an assistant professor in the department of psychology at Durham University. “But what we consider important does not necessarily mean that these are the things we need.”
If you ask people which sense they wouldn’t be able to live with, most will say sight, but evidence suggests what we’d really miss is our sense of smell. “There is a much higher rate of suicide and suicidal thoughts among people with anosmia, because it is a feeling that is so strongly linked to our emotions,” says Scheller.
Does the neglect of some senses in favor of others affect our emotional lives? Insofar as our emotional health is linked to our social health, the answer is almost certainly yes. “Smell is a very important cue for social communication and it’s something that isn’t implemented in any technology we use today,” says Scheller.
For example, we have been shown to be prone to it sniff our palms unconsciously after shaking hands with someone. “It gives you hints about all kinds of things, from their health, to their age, even their personality,” says Spence. “A fair amount of that is lost if we only interact digitally.”
Touch is equally important to our emotional lives, and in ways that the finger-focused haptics of our digital devices cannot satisfy. C-tactile afferents, a type of nerve receptor abundant on the hairy skin of our arms (but not the pads of our fingers), have been shown to create positive emotions when stimulated. “These receptors like slow, warm, tactile caress,” says Spence.
The cold, sleek touchscreen of a smartphone simply cannot replace the soft, warm, imperceptibly smelly skin of another human being. For adults, this may mean less satisfying social lives, but for a generation of children increasingly socialized by technology, the consequences can be severe.
Scheller says children learn to interpret their senses with reference to each other. We can learn to associate a subtle smell with the sound of a person screaming or the sight of them smiling and use these signals to navigate social situations in the future. “Those kids who grow up with less input basically have less training to be able to categorize what certain things smell like, or what a certain touch might mean,” says Scheller. “If we suddenly take away something that has evolved over millions of years, it won’t just be the removal of one sense, but it will affect how all the other senses work.”
Marianna Obrist, professor of multisensory interfaces at University College London, says: “The way we experience everyday life is for all our senses. Everything is multisensory.”
For example, it is easy to think that the experience of eating is primarily about taste, but our food’s shape and color, smell and sizzle, temperature, texture and weight appeal to our vision, smell, audition and touch. “All those senses have already started playing before you even eat,” says Obrist. And then there is mouthfeel: the physical sensations of spiciness or sourness and of course the flavor.
Removing just one of those senses can impact the entire experience. For example, when people eat ice cream in the dark they are less likely to enjoy it, or even be sure what it tastes like. “When we have multisensory stimulation, we get a much better and richer representation of the environment around us,” says Scheller.
So what are we doing to make our technology more multisensory? Obrist previously led SenseX, an EU-funded project that aims to help designers come up with new ways to integrate touch, smell and taste into their products. The team’s efforts included spraying scents under a subject’s nose to heighten key moments from Christopher Nolan’s film Interstellar, blast them with ultrasound waves to simulate touch and used high-intensity acoustics to levitate food on the tongue without the need for wires or tubes.
It’s hard to imagine you’ll be watching anytime soon Robert Duvall’s Lt Col Kilgore delivers Apocalypse Now‘s most famous line while your laptop sprays eau de napalm-in-the-morning up your nose, but smell and taste interfaces may be on the horizon. Researchers already are use of AI to try to find primary odors from which any smell can be spoiled, and Obrist is the chief scientific officer of OWidgets, a company that makes digitally controlled odor delivery systems with applications in research, healthcare and immersive reality experiences.
There are also companies like Dexta Robotics in China that bring tactility to virtual reality with a glove that it the Dexmo.
to newsletter promotion
“Dexmo can provide tactile feedback and force feedback at the same time,” says Aler Gu, CEO of Dexta, “which means when you run your fingers through a virtual brick, you can feel the texture of the surface. When you grab and move the brick from one point to another, you can feel the physical shape.”
Media that engages all the senses will certainly enrich our daily interactions with technology, but it’s not hard to imagine more insidious uses emerging. In 1957, an American market researcher named James Vicary claimed to have merged single frames reading “Eat Popcorn” and “Drink Coca-Cola” into a film. He reported a 57.5% and 18.1% increase in popcorn and Coca-Cola sales respectively, and the concept of subliminal advertising was born.
Vicary was later exposed as a fraud and the effectiveness of subliminal advertising was a matter of debate since then, but would technology that can digitally deliver smells and tastes be a gift to unscrupulous advertisers? “Our bodies have a very strong emotional response to [these senses]. They can be extremely powerful,” says Scheller. “It has great potential to influence our decisions because we are very emotional decision makers.”
Studies have shown that exposure to certain tastes and smells can influence our judgment of other people’s appearance and personality, and even change our behavior. Taste bitter food, for example, can make us hostileand a 2005 patent application suggests that the smell of pink grapefruit will make a man see that a woman is younger than her actual age.
Obrist’s team found it sour taste can make us more willing to engage in risky behavior. “You might be doing e-banking or online shopping, and you’re drinking your lemon drink, and that can indirectly influence your decisions,” she says, and it’s not hard to imagine how an e-commerce or gaming app could exploit what tastes and smells can produce.
To some extent this sort of thing is already happening. Companies are known for pumping pleasant flavors into their stores, and the American chain Cinnabon deliberately place ovens near store entrancessometimes trays of just sugar and cinnamon, to attract passing shoppers.
And what if we take it even further? Of the nearly 63 million people who voted for Donald Trump in 2016, the vast majority only experienced him through two of their senses. What if the media used our devices to deliver a subtle flavor of sour milk while broadcasting a speech from one political candidate and freshly baked cookies for another?
After all, a study from 1940 showed that people were significantly more or less likely to identify with political slogans such as “No to war and fascism!”, “Workers of the world unite!” and “America for Americans!” depending on whether they were subjected to a foul smell or given a free lunch.
If the news allowed us and our leaders to taste air pollution in Delhi, feel wildfires in California, or smell the smoke and sewage in Gaza, appealing to our more emotional senses would move us to act, or to bury heads deeper in the sand? It’s hard to imagine a willing audience for such a sensory assault, but our senses evolved to help us navigate and respond to the world we live in, and from that point of view, only two of them can’t be ideal don’t be “The more information we have,” says Scheller, “the more able we are to actually act within our environment.”
For now, instead of holding out for digital technologies that can stimulate our neglected senses, Scheller suggests we might do well to go outside and see our friends in person, feel the breeze on our skin, smell the roses . After all, as far as our devices are concerned, we haven’t smelled anything yet.
