larger language models are revolutionizing business, daily life, and the military. However, during several conferences on AI, we heard speakers saying, We don't know exactly how these models work, or things like, We don't know exactly how neuron networks deliver their output. The lack of understanding might not be a significant issue if the outcome is useful and not dangerous, but can we truly trust these models 100% in domains like defense, security, health, finance, and other high-stakes situations? Probably not. Fortunately, researchers are working on understanding the internal mechanism, such as which neurons trigger specific behaviors and which connections are activated. One approach to achieving this understanding is to reverse engineer the algorithms implemented by neural networks into mechanisms comprehensible to humans. The field is called mechanistic interpretability. In the past two years, mechanistic interpretability has seen rapid progress. For example, researchers have used new techniques to discover how large language models implement particular behaviors. One of these researchers is Dario Amodei, an American-Italian engineer who has worked as a research executive.

at BYU, Google, and OpenAI in 2021. Dio left OpenAI to start Anthropic, an AI Safety Research company. Dio Amod was at Vivatech last summer and was on stage to announce a recent breakthrough. His announcement and comments are the best introduction to the discipline of mechanistic interpretability. Take a listen. Um, you know, today's modern AI systems are, you know, made from these large neural nets that have hundreds of billions of parameters, um, you know, hundreds of millions of neurons, and it's very hard to understand what's going on, uh, inside them. So, since almost the founding of Anthropic, one of our co-founders, Chris Ola, um, uh, helped to invent the field of what's called mechanistic interpretability, which is trying to look inside the mathematical behavior of the neural nets and understand what's going on inside them. And so, we've had a research program on this running for the last 3 and a half years. It was pure research up until now, I, and so the research that we put out

yesterday was we showed in one of our production-grade models, Claude 3 sonnet, which is the middle of our three models. We're able to look inside the model and find 30 million what we call features, which are things that correspond to concepts, and we can watch them light up as the model reads content and generates content. So, an example of this is there's a feature that corresponds to when an actor breaks the fourth wall and says something, something that makes them, you know, that shows that they're aware they're in a fictional setting. And if you ask Claude, our AI model, about itself, that feature will light up as it starts to talk about itself as an AI system. And we can turn on and off that feature, and when we turn on and off that feature, Claude's behavior changes. There are features for security vulnerabilities in code, there are features for bias for or against various kinds of groups, there are features that correspond to different genres of music. So, we're starting to see a whole atlas of what's going on inside these models, and we're

able to intervene into them and change their behavior. On the stage of Vivatech, the guest didn't speak too much about the tools used in mechanistic interpretability. Instead, the focus was on some of the more common negative outputs of LLMs, like hallucinations, something that at Anthropic were able to identify. There also were features for deception, uh, there are features for incorrect facts, um, various things like that. Now, when models hallucinate, they don't always know that what they're saying is incorrect, just as, you know, humans make a distinction between lying and simply being misinformed. Uh, so I don't think this is the only source, but reducing hallucinations is, I think, one important application of this discovery and hopefully other discoveries that will follow it. Another issue that is not often mentioned is that of psychophancy. This is when a language model shows an attitude to completely agree with or praise the user, regardless of whether the user statements or opinions are correct or reasonable. Needless to say, this comes at the expense of providing accurate or honest information.

Researchers attribute a psychopathy to the training data, which often includes polite, agreeable language, and to reinforcement learning approaches that may unintentionally reward agreeable behavior. Understanding the internal mechanism that generates psychopathic responses may help researchers modify this mechanism. Amod gives the best rendition of the phenomenon. There's a salient feature, so when you turn this feature up, the model will give just completely over-the-top praise. Like, if you ask it, How are you? it'll say, You are so thoughtful for asking me how I am. You're the greatest person in the world. And, you know, just crazy over-the-top praise. So, this causality, this ability to intervene, you know, that's the ultimate test of whether you understand systems, and we have a long way to go. For those interested in learning how mechanistic interpretability actually works, Chris Hola, co-founder of Anthropic, and that was mentioned by Dar Amod, has several introductory videos. Right after Olaf was also

a speaker at a workshop in Vienna where the goal was to bring together academia and business to come up with standards, goals, and a common language to carry out research in this domain. From the workshop, we learned which techniques are used in reverse engineering and neural networks. Here is a quick overview: circuit analysis, the goal here is to identify and study specific parts of neural networks that perform particular functions; activation patching and causal interventions, with this technique, specific parts of a network are modified to see how it affects behavior; superposition analysis, studying how networks represent multiple features in the same space; sparse autoencoders, this is a method for decoding superimposed representations; qualitative analysis of network components, a technique used to study individual neurons, attention heads, etc.; scaling and automation techniques, these are the efforts to make interpretability methods more efficient and applicable to larger models. One thing that interpretative mechanics cannot do is to eliminate or limit the capabilities to create deep fakes or writing content emulating the style of celebrities. On this matter, Amod does not

have a solution, but he thinks the society will adopt ways to deal with these issues. Take a listen. You know, it kind of comes back to have you trained on the data that kind of enables you to do that, right? And these things are a little bit fuzzy, right? Because you can, you know, even if the model hasn't read content by a particular person, right? We're kind of building general cognitive tools, right? So, I think in the long run, we're really going to need to grapple with the idea that as these AI systems become very smart and capable, they're going to be capable via all kinds of routes of kind of doing these feats that infringe uncomfortably on what humans are able to do, not even specific humans. And so, you know, we're going to need to think about as a society what our rules and thinking about this are, right? It's, I want us to think big in terms

of how society is organized because AI is a very broad technology. If companies like Anthropic will be able to unlock the secret of the black boxes in LLMs and are able to correct potential errors like those that lead to hallucinations and psychopathy, we can expect a future in which AI will be infallible and know more than humans in every domain. If this is true, what domain should we study today that is AI-proof and in which professional humans can still have a voice? Has two different opinions, one for the short term and one for the long term. Take a listen. So, I would again separate it into the long term and the short term, right? In the short term, I am confident that areas like biology, areas like robotics, there's just a huge amount of human expertise and know-how that, you know, today's language models may know something about the field, but, you know, being able to actually slot it in, there's going to be a lot of human expertise and human structures that are required.

I'm very confident of that in the short run. I think in the short term, we have this strong feeling that AI will be complimentary to humans. So, um, there's this um, economics researcher, I think you know him, Eric Bolson, who's done these studies that show that by default, folks who try to apply AI, like downstream customers, will often apply them in ways that by default, uh, replace humans. But if they're more thoughtful about it, if they find a way for AI to complement humans, that ends up increasing productivity more so. We very much want to encourage these complimentary applications. They're both good for jobs and for humans, and they end up being more productive. I I think in the long run, again, it ties into this bigger question of will there be a time when AI systems are better than most humans at most things, or even better than all humans at all things. And I think once that happens, we're all in the same boat, right? We're all in the same boat.

same position, you know, like whether you're a manual laborer or the CEO of a company or, you know, or an actor or actress. It's the same thing. We're all going to be in the same boat, and so we're going to need to rethink as a society, you know, how we deal with that and how that works in the long term. If we reach this world where AI systems are as good at most or all things, you know, then we really need to think about that, right? Like, we've had a long era, the industrial era, during which people derived their value from what they're able to produce economically. And I think in the long run, on longer time scales, AI may challenge that assumption, just as the Industrial Revolution, you know, challenged the social structure of feudalism, and farming challenged the, you know, social structure of hunter-gathering tribes. So, I don't know what the next thing in that sequence is, but I think we do need to

think start thinking about what Dario amod also touched upon two topics that were the object of previous videos on this channel: open versus closed large language models and data becoming less relevant. You can watch the videos at the link above. So again, I would talk short-term, long-term. So I think, you know, in the short term, in terms of safety and security concerns, I'm not concerned about any of today's open models. I'm not concerned about, you know, open models that come out of the next year. I'm not concerned about closed models that come out of the next year either. Um, uh, but you know, I think in the long term, you know, we have various safety and security concerns about models in the long term. But I think this is more about powerful models and future models versus today's models than it is open versus closed models. And in fact, I think open and closed models are becoming more like each other. So, you know, on one hand, I think we've seen this pattern where when companies or efforts are getting started, they'll often do

open weights models, and they switch to close Source models as they gain traction, as they have products at the same time. Close Source models are increasingly offering things like fine-tuning customization, many of the benefits that you get from open-source models. For example, we're planning to offer such benefits through Amazon AWS, right? Similarly, clouds are starting to increasingly host open weights models. You have to pay for the cloud service; there may be some fee that's charged on top of it. So, in many ways, they're converging to things that are very similar to each other. Many kinds of data are overrated because increasingly, not just us, but other AI companies are starting to get good at creating synthetic data. So, I think data will generally not be a bottleneck for specific applications. Particular types of data will be important, right? The ratio of synthetic to real can't be infinite, but in terms of the models we want to scale for the foreseeable future, it's hard to say.

for sure, but synthetic data is looking very promising. I think the thing that'll determine who has the best model is talent, invention of the best architectures, use of the data and compute that we have, as well as we can possibly use that compute. I think in any industry, it almost always comes down to talent. [Music]

Anthropic: Building LLMs that You Can Trust.

Anthropic: Building LLMs that You Can Trust.

스크립트