I am a bear. So says AI.
I scroll down the lengthy report on my iPad and I smile. I am a watchful, steady guardian; a person of quiet, formidable substance; and I have a broad, square-angle jawline that means I commit long term. I’m already feeling pretty good about myself.Ěý
I then see that my face has a historical parallel: Dwight Eisenhower. Not sure about that one, but just below him are other faces like mine: Theodore Roosevelt, Ernest Hemingway, John Goodman, and Adele.
Not bad company.
All I had to do was take a photo of myself with a neutral expression on my face, and the app used some version of artificial intelligence to analyze it. It took measurements, found tiny deviations from the norm, and told me what my “outside” revealed about my “inside.”
This may look like a harmless, 21st-century take on astrology. Let the kids play with it on their phones and get told they have a “choleric-melancholic temperament” or a “Saturn-Mars” face shape—whatever that means.
But this app is just the tip of a menacing iceberg. Physiognomy, the rank pseudoscience that fell apart alongside the Nazis, is back, courtesy of artificial intelligence. It promises to surveil employees at work, identify terrorists in a crowd, and figure out who’s gay and who’s not from a single photo.
It doesn’t need to be accurate; it just has toĚýlook like it works,Ěýand bad research is already making sure of that.
Welcome to the age of neo-physiognomy.
“Do you wanna know how I got these scars?”
The problem with physiognomy is that it is deeply ingrained in our brain. The word comes from the Greek for “judging a person’s nature by their features,” and if you have ever spotted a movie villain with a giant scar, you have seen physiognomy in action. Its core idea is seductive: that what is inside a person manifests on their body. Beautiful people must be good and good people must be beautiful. We can trace this notion all the way back to ancient Mesopotamia.
In the late 1700s, physiognomy captured mass attention through the work of the Swiss writer and theologian Johann Lavater. The books that spread this nonsense were replete with hand-drawn portraits that we would now judge as grotesque, insensitive, and just plain wrong. You can scroll through a digitized version ofĚýĚýto “learn” all sorts of misguided suppositions: that a nose that droops is evidence of deceit; that an ear that’s a bit squared off at the top signals honesty; and that you can tell that your husband will be unreliable by the shape of his head.
If you are reminded of phrenology—the pseudoscience that claims bumps on the head correspond to distinct personality traits and aptitudes—you’re not wrong, as phrenology and physiognomy are conjoined twins. Many of the major physiognomists were also phrenologists.
When you think, wrongly, that you can tell a person’s character by looking at their face, you’re on a path straight to eugenics. The father of criminal anthropology, Cesare Lombroso, was telling whoever would listen that you could just see who would become a criminal: their faces would express traits from our caveman ancestors. Armed with this criminality detector, it’s no wonder that considerable harm was done in order to improve human stock. This thinking culminated in Adolf Hitler’s German Reich, before being swept off the map in its aftermath.
Yet, physiognomy never died; it hibernated. Gut-level intuitions about why people behave a certain way can never be eradicated by science: case in point, astrology is still with us today despite having been shown,Ěýtime and time again, not to work.
It was artificial intelligence that awakened physiognomy from its slumber. The people training neural networks—computers that mimic the way our brain cells work and that are promoted under the umbrella of “artificial intelligence”—admitted that, sure, the human eye can’t tell if a person is good or bad by glancing at their face. It simply lacks the precision.
What if a computer could?
Parsimonious explanations
ĚýWhen you trawl pre-print archives and conference proceedings, and even the peer-reviewed scientific literature, you will find write-ups by computer scientists—AI wranglers, if you will—that gush about what their neural network managed to do.
These wranglers feed their AI two sets of photos and train it to identify why set 1 is different from set 2. The sets could be gay men versus heterosexual men; they could be people with depression versus people without it; criminals versus non-criminals. At first glance, the people photographed in set 1 should not have anything visible that is different from those in set 2; but the AI finds something, or a cluster of things, and it’s able to separate people into their respective sets at a higher rate than if you were to merely flip a coin.
This looks like proof that physiognomy is real and that AI can somehow read people’s faces in a way that we just can’t. Are you autistic? AI can tell. Are you a lesbian? AI can tell. Have you been convicted of a criminal offense? AI can tell.
But as was pointed out inĚýĚýby authors whose affiliations include the Mozilla Foundation and Google Research, this reanimation of the pseudoscience of physiognomy is built on a mirage. If you test 18 personality factors in two small enough sets of photos, your AI will do better than average for a couple of them by sheer luck. And that’s just the start.
AĚýĚýon using AI to identify people with autism displayed its fatal flaw proudly by publishing sample images it had used. From looking at the three people classified as “autistic,” it is clear to me (and to the authors of the takedown) that these men have a chromosomal anomaly, like trisomy 21. They may also be autistic, but that’s unlikely to be what the neural network is seeing in their faces.
From that same paper, the scientists also fed “normal” images into their AI—the three presented in their paper being studio photographs of models—as well as images tagged as “drug addict”—mugshots of people being arrested for a crime, including Tiger Woods. This problem also affects aĚýĚýfrom George Mason University, now retracted due to a failure to seek ethics approval, where criminality can be detected in the face... so long as the criminals and non-criminals had their photos taken in different circumstances. The authors cop to it in their conclusion. Is their neural network truly picking up the seed of criminality in someone’s face—an idea that runs against what we know of human biology—or is it instead noticing differences in lighting, framing, distance from the camera, and background? The latter, assuredly.
Besides, when we feed photos of criminals to an AI, are we also including white-collar criminals, who wear button-up shirts and suits to go to work and embezzle money? Or are we biasing our artificial model in the same way that our legal system is biased?
In this way, we end up laundering humanity’s prejudices, since computers are promoted as objective and thus free of our silly biases. Once technology is elevated in this way, the real harm can begin.
“Thoughtcrime was not a thing that could be concealed for ever”
The papers I have read—all significantly flawed and many fatally so—open the way for the unfurling of prejudicial surveillance technology that should raise significant alarm.
Israeli startup FaceptionĚýĚýAncient Chinese face mapping and epigenetics to justify selling an AI that scans faces in videos to identify who has a high IQ, who is a professional poker player, andĚý. As we all know, terrorism is an inborn trait, much like blue eyes and dark skin. Company HireVue, meanwhile, tackles the job interviews that are now done remotely in order to provide employers with a highly questionable report on what the candidate’s face betrayed—after all, as the authors ofĚýĚýdisturbingly put it, one of the most significant tasks when conducting remote interviews is “deception detection.” Once hired, this candidate might be subjected to a technology discussed atĚýĚýwhere a webcam is parked in front of a worker’s face throughout the day and an AI detects their emotions in real time so that, the authors concluded, “the employer can make [sic] the better decisions about the well-being of their employees.”Ěý
Some of the scientists who publish their findings appear to be doing so to warn others. TheĚýĚýon AI identifying gay men and women based on their faces, while erasing bisexuality from the equation, contains a note: large queer organizations were alerted by the authors, prior to publication, that the results of their study would be released shortly. Others lean into the criminalization of certain behaviours. AĚýĚýsupported byĚýĚýreports on the training of an AI to recognize people who beg on the street, citing the need for public safety. The authors define beggars as “people who annoy or harass surrounding people by begging or asking for money.” Does police need to be alerted via AI to the presence of people begging for money now?
A bear becomes a horse when you smile
In aĚý, authors Luke Stark and Jevan Hutson put it best when they named this resurgence of physiognomy via AI “the automation of the caliper.” I had the privilege of once visiting a room of skulls at a university in Scotland, closed to the public: a small, chapel-like chamber lined with bookcases filled with human skulls, three deep, from all around the world. There are lessons in anatomy to be learned here, but this kind of collection is also the sort that felt the stroke of a pair of calipers, trying to measure if Black skulls were smaller than white ones.
Physiognomy is untrue. Yes, beautiful people might receive more attention from others, which could lead them to become more extroverted, and thus a minimal association between conventional beauty and being a social butterfly could be found. Scientists have also hypothesized that sexual orientation depends on hormone exposure in the womb, which might also affect how the face develops. But even if this is true, technological applications rely on a binary (gay versus straight) instead of the very real and messy spectrum of sexual orientation.
It’s not rare to find a seed of truth inside any pseudoscience; but by and large, you cannot read faces like books and spot the swindlers, the heroes, the men with boyfriends, and the women who will one day end up in jail. DNA is not destiny; it is predisposition. Your fate cannot be translated from your face by the naked eye or by a sophisticated computer. But the shoddy studies coming out of the woodwork are crowning this pseudoscience anew, and I worry about the near-future consequences.
The app I used to analyze my face—and there are others; mine was free but others require payment—appears weighted toward giving you a pleasing interpretation (much like astrology), and it certainlyĚýlooksĚýlike it works. I uploaded a photo of dictator Augusto Pinochet, and the app told me his animal was the bull, “the immovable anchor of authority.” Maybe it was written on his face. Maybe his military uniform tipped off the AI.
I wasn’t happy being a bear, though, so I took a photo of my face again: same clothes, same background, same lighting. But this time, instead of adopting a neutral expression, I smiled.
My bear became a horse. My choleric-melancholic temperament morphed into a sanguine-phlegmatic predisposition. My once “deep-set eyes”—they guarded my emotions, apparently—were now “steady” and “focused.” I was still like Eisenhower, but also like Adam Driver, Sarah Jessica Parker, Julia Roberts, and Céline Dion.
The book of my life had changed because I made my page smile.
Would you rely on this?
Take-home message:
- Physiognomy is a pseudoscience that claims who a person is and what they will become is expressed on their face, and it is closely related to phrenology, the misguided study of bumps on the head to identify what a person is good at.
- A subset of research into artificial intelligence is devoted to bringing physiognomy back by training AI on photos so that it can identify who has depression, who is gay, who is autistic, and who is a criminal or a terrorist.
- None of this research stands up to scrutiny, as often the two sets of photos (e.g. criminals vs. non-criminals) are very different in other ways, and that is likely what the AI is picking up on.
