Image from: https://commons.wikimedia.org/wiki/File:Sonnenblume_Nahaufnahme_H%C3%BCllkelch.JPG
Last week, Jason Brooks introduced a deceptively simple idea – the curse of knowledge – and showed how it shapes not just how we communicate with each other, but how we communicate with AI. This week, he goes deeper. If prompting is where the curse of knowledge shows up in practice, synthetic data is where it shows up in the results. As AI-generated respondents become more common across research and strategy, the question of what we actually know and what we only think we know has never been more consequential. Jason brings his lens as a cognitive scientist to one of the most seductive promises in our field: that more synthetic voices add up to more understanding. What he finds is something more complicated, and more important.
There’s been a crime! And as investigators, we’ve got to get to the bottom of what happened. Good news: the crime took place in public, in broad daylight, and there were tons of witnesses. Bad news: our chief says we don’t have the time or resources to go out and interview them. Instead, the department has commissioned the service of a special investigator: Chad G.P. Thompson. Chad seems super smart, is super cheap, and works super quickly. Because we know where and when the crime was committed, we have a pretty good idea of the types of people that were likely around to be witnesses and because Chad has spent a lot of time talking to those types of people before, Chad is confident in being able to impersonate the witnesses for our interviews. “I’ll be Rebecca, a 36-year-old middle-class mother of two, what would you like to know?” says Chad during our interrogation. This is a little weird, but we ask, “can you tell us what you saw on the afternoon of the crime?” And Chad—I mean “Rebecca”—proceeds to elaborate about what “she” was doing and saw during the moment of interest. “And now, I’ll be Reggie, a 24-year-old single graduate student at the local dental school…” Chad says, pivoting in seat and providing a new elaboration of what Chad—I mean “Reggie”—saw and did during the moment of interest. This goes on a few dozen more times and within mere minutes, we have a vivid portrait of a crime scene. The stories all seem very plausible, confirming many of our intuitions about what the experience must have been like for a 36-year-old middle-class mother of two or for a 24-year-old single dental school student. And we have DOZENS of these stories. “Surely we’ve got a rock-solid case,” the chief asks, “right?”
You see the problem, of course. We’ve learned nothing about what happened. We’ve learned nothing about the experience of even one real person. All we have is caricatures—even if they feel like really rich and plausible caricatures.
“Ok fine,” says chief, hearing our concerns, “I’ll free up some budget so you can go out and interview two real witnesses, but that’s it!” So we go out and interview two real witnesses. Surprisingly, they each give us different accounts of what happened and the chief thinks having just two witness accounts that conflict will be a problem for convincing a jury about what happened, so another special investigator is brought in to help with the case. The investigator is from France: Détective Cláude. Cláude has developed an interesting method that listens to our real witness interviews and then not only pretends to be those witnesses if we have follow up questions to ask, but also pretends to be other people that were sort of similar to the two we really interviewed. So now we’ve got pages and pages of testimony from dozens of “witnesses”, including a few pages of testimony from a few real people. The chief hails Détective Cláude a hero, “with all that testimony and from all those different witnesses, this will surely convince a jury about what happened at the scene of the crime, right? RIGHT?!”
Maybe that could convince a jury, maybe not, but would it convince you?
We can take the testimony of the two real witnesses for what it is: imperfect but genuine evidence of an experience. The additional testimony, whether generated from whole cloth (like Chad did) or modeled from those two witnesses (like Cláude did), adds no new observations. Whether the technology impersonates 100 witnesses, or 1000 witnesses, or heck even one TRILLION witnesses (that would be more than all the humans who have ever existed), our confidence about what happened wouldn’t increase. It may create the appearance of certainty, but it does not increase our knowledge of what actually occurred. That’s the illusion of precision.
This is not, however, to discount the utility of a clever, informed and proven mind.
For knowledge can take different forms. There are literally thousands of years of philosophical writing from all over the world about the different ways of knowing (what fancy philosophy people call “epistemology”, a kind of forebear to cognitive science). Hindu philosophy, among other traditions, distinguishes between Observation (seeing something), Inference (using reason), and Intuition (sensing something, based on wisdom). We can relate these ways of knowing to our industry: When we conduct market research (quantitative surveys, qualitative interviews, behavioral tracking, or web scraping), we’re acquiring knowledge about the real world from our observations of real people. When we apply statistical principles and predictive modeling to make claims about how the world works beyond what we’ve directly observed, we’re using a form of inference. And when we apply judgment to all that we’ve observed or modeled, selecting what is and isn’t worth paying attention to and acting upon, based on our entire lived experience as both a human in the world and a professional in our field, we’re using a form of intuition.
At a foundational level, it’s important to understand what’s going on with real people in the real world if we wish to advise businesses on how best to get real people to exchange real money for real products and services--especially when the risk of failure is high. But there are also situations where we might rely more on intuition—wisdom—from trusted, experienced experts with a track record of success. Because in a perfect world, we would have robust observations about the world we wish to make decisions within, but in practice, not every business decision gets the luxury of being data driven. When a seasoned executive or prognosticator decrees that a new product might be worth pursuing (or not) based on how they intuit consumers will think and behave, they may very well be relying on their accumulated experience and wisdom forged and refined in a lifetime of success and failure—and they may very well be right often enough to justify acting on their judgment. But at the same time, it would be silly to pretend as though the intuition of an oft-right expert is the same thing as observing what real people in the real world think and do. And when the risk creeps high enough, even many a self-confident expert would prefer to base their decisions on real world observations rather than just intuition alone.
When AI tools are used to generate synthetic respondents or synthetic responses, they’re essentially generating a form of intuitional knowledge. We might be tempted to think of it as just inference (modeling), but the vast amount of information AI technology is built upon and the vastly complicated ways in which the technology learns from that information (often inscrutable even to the engineers that build the technology), means it operates less like a straightforward statistical model with clearly defined inputs and equations that convert them into outputs, and more like the vastly complicated, vastly but imperfectly informed, and mushy ways our brains operate (partly why they can seem so human to us), and thus, it can be argued, they are more a kind of intuition. And the problem with intuition about reality, like any model, is it can be wrong.
We can easily recognize this when thinking about legal crime cases. It’s not enough to convict someone of a crime just because someone has an intuition about what could have happened—we need to see evidence of what did happen (observations from the real world or testimony from real people about what they experienced in the real world). It’s not enough for our hypothetical Chad or Cláude to pretend to be people it thinks might have witnessed a crime in order to deprive a suspect of their freedom for having committed the crime, thank goodness—the risk (in this case, to the liberty of an innocent person) is too high.
But intuitions can still be useful in certain situations, especially if they come from a well-informed, clever mind with a track record of being right. Useful because they’re formed quickly, cheaply (well, depending on who’s charging), and can take inspiration from a wide variety of disparate things. When the cost of failure isn’t too high and the cost of waiting too long for good observations is, intuition sourced from a well-vetted person (or system) might just be good enough. In theory, the best synthetic respondent or synthetic response technologies can provide this form of useful intuitional knowledge. They’re informed from prior observations, their judgments refined repeatedly over trials of success and failure, and rely upon a foundation of nearly unfathomable technological complexity and resources, and they have a track record of prediction that has been vetted (hopefully). They just can’t be treated like observations of reality, and treating them as such when the cost of failure is high, is a recipe for future disaster: for missing a critical finding, a novel signal or interaction, a sign of change, a nuance coded within a chain of answers that real people give to a set of our questions based on their real experiences in the real, ever-changing world that will inform a consequential business decision. So it can be fine for generating and/or challenging early hypotheses (like how one might run ideas by a handful of people to get their take) when precision isn’t as important, but testing hypotheses when there are big consequences on the line—is that a time to give in to the illusion of precision?
In some respects, what the best AI systems do resemble something humans have always done. In 1949, Joseph Campbell wrote, “The Hero with a Thousand Faces”, a landmark book about the power of myths and how common themes in stories have been told and retold in different forms across different cultures, across the sweep of human time. Whether the hero of the story is Buddha, Horus, Moses, Odysseus, Luke Skywalker, Frodo Baggins, Harry Potter, or Spiderman, the stories exhibit similar features about the characters, the world around them, and the struggles they face. These stories are retold over and over again in different forms relevant to the styles and trends of the day because they tap into something deep and meaningful about the human condition. They can confer wisdom, shaping our own intuitions, and the stories have survived through time not because they’ve been precise about actual events, but because they have been useful to us (we don’t hear much of the ancient stories that weren’t useful to know). The AI technologies that harvest so much of the collective writings of humanity exists in a similar way—where at best, they can confer a kind of intuitional knowledge to us about the human condition—which can be very useful to us when we have a decision to make, and limited time and resources to go out into the real world and learn something about what is actually going on before we have to make the decision. But like mythological stories, their usefulness to us is in how they influence our thinking, not in providing a precise picture of our real world. It’s just that many time, we also need that precise picture.
And while the hero of the human story may have a thousand faces, and the AI-hero of your business story might provide some useful insight to you in a pinch, don’t let its portrayal of a thousand faces fool you into believing you’ve gained the evidence of a thousand real voices. Should a thousand synthetic faces convince a jury? Should they convince you?
