Chatbots Say the Darndest Things

Although programs like ChatGPT are certainly impressive, shortcomings such as bias and hallucinations demand our attention, Belinkov says. But exactly why a neural network gives one particular output rather than another has always been rather opaque. While the overall architecture of such networks is well understood, the actual cause-and-effect links from input to output can seem deeply mysterious. In a neural net, data points are represented by abstract mathematical quantities known as “vectors,” and “the whole process happens in some high-dimensional vector space, which is very hard for people to interpret,” Belinkov says. “It’s not a two-dimensional or even three-dimensional space that we can understand. It’s maybe 1,000 dimensions.” As a result, he adds, “we don’t really know what’s going on there. In this sense, it’s really a black box.”

With no way to track in detail what that vast array of vectors is actually doing, how can one tackle problems like bias in chatbots and similar AI systems? Belinkov’s strategy is to probe the network’s internal structure, examining which parts of a network are active at each stage of the process. He describes the process as an “intervention” in which one runs the program multiple times, each time tweaking one, or a few, components. “And every once in a while, you find there’s a neuron” — he uses the same term used to refer to cells in the brain — “where, when you switch it off, the machine gets lost. And that tells you, OK, maybe that’s where things are happening.”

Belinkov draws a comparison to an old-time arcade game in which a ball is inserted at the top of a wide, flat box with an array of pegs in it that allow the ball to move along some paths but not others. The user can remove or insert various pegs in order to try to steer the ball one way or another. Often, he says, many of the pegs — or in the case of a chatbot, network connections — don’t actually affect the end result.

“We have to understand where, in the system, there’s a switch — maybe one or more switches — that determines what it will say,” explains Belinkov. Only a few of these switches would actually determine what text the system produces in response to a particular input. “So, this question of ‘why’ is, I think, very much related to the question of ‘where,’” he says. “If we can pinpoint one switch, or multiple switches, that are responsible for producing a particular output, then we have an answer of why.”

The fact that some parts of the network seem to be much more important than others comes as a surprise, Belinkov says, given how interconnected the whole setup is. “If everything is connected to everything, then computation should be distributed and information should be likewise distributed. But it turns out that it’s not. For certain inputs, certain neurons activate much more than everything else.”

Another potential strategy is to feed the system new inputs, but that’s not as easy as it sounds, cautions Belinkov. For example, if an LLM was trained two years ago, some of its outputs will be out of date — but retraining it on the entire internet could cost millions of dollars. So, a more efficient strategy is to just tweak the specific parts of the system that have been implicated in producing problematic outputs, Belinkov explains. To combat a problem like gender bias, then, a better approach, he says, is “to look at the internal structure, identify the roles of different components, and then change them or edit them in a way that we think is desirable.”

David Bau, a computer scientist at Northeastern University in Boston who has collaborated with Belinkov, notes that the remarkable capabilities of large language models “have been a genuine surprise, even among the experts of the field.” Belinkov’s research is important, he says, because the opacity of LLMs makes traditional debugging techniques ineffective. “Yonatan’s research about these models’ internal mechanisms will be critical in closing our gaps in understanding. Already, his work is laying the groundwork to help identify and solve potential problems of bias, robustness, misinformation and privacy that can emerge in large models.”

Studying the internal architecture of AI systems may sound like an esoteric pursuit, but with AI systems playing an ever-increasing role in society, the stakes are high. When computers were first invented some 75 years ago, the assumption was that they would help us. And in many ways they have. But to continue to aid humankind, the objectives of AI systems have to be aligned with our own. Indeed, Australian philosopher Toby Ord has estimated there’s a one-in-ten chance that “non-aligned” AI will trigger a catastrophe during the next century.

Belinkov admits that for much of his career he considered the “alignment” problem to be a low priority, something to worry about in the far future. The threat “seemed so far-fetched and so remote,” he says. “But I have to say that, recently, I’ve started thinking that, yes, maybe it is time for both scientists and legislators to start thinking about the alignment problem more seriously. We need to make sure that AI systems are not biased and perform what we want them to perform. But at the same time, we should think about longer-term risks.”