Keynote 2: Breakthroughs in LLM research and Constitutional AI
Gen AI Summit 2023 - Open or Closed
Anthropic: Jared Kaplan, Cofounder and Chief Scientist
SUMMARY KEYWORDS
systems, ai, models, train, gpt, plot, customers, human, generative, scaling, behavior, language, lawyers, trends, interesting, early, number, work, computation, contracts
SPEAKERS
Jared Kaplan
Jared Kaplan 00:00
As you may have the better play but hopefully getting more people will be playing soon. So I'll be talking a little bit about the trends that we see driving AI progress or why we're expected to get even more exciting, maybe maybe a little bit scary. So there are two things that you need to continue to make progress in AI systems. There is a exponential trend in using more compute power to make things better and there's a similar trend in algorithmic environment for making these systems more and more efficient. And that's really driving what you see. That's possible this year. That wasn't possible a couple years ago, and why I think the future is very interesting. The other thing you need for these systems and in generative AI systems, you don't want them to just be ignored edge, you your lesson to the snakes you need some kind of stability or reliability or safety work to ensure that these systems behave. So anthropic really aims to keep building some of the most powerful volunteer existence in the world while focusing a lot of our research effort really, really early on making these systems more secure and more reliable, and safe. And we really think that AI is probably most people believe the answer here is going to permeate the economy and grow in importance over the next few years. So who are we and maybe I'll actually give you a slightly longer history lesson since we're at the Computer History Museum. So it looks to the prospect of being very our CEO. We're in some sense the team that built GPT three, you can see the original GPT three paper. Daria was the director of the project, the lead engineers, now running, staring at it, love it. But it's sort of interesting to ask, since we're here, where did these systems actually come from? GPT three What about JPEG one? GPT? Two, and so the the interesting person who is on this paper, but not highly as much as Alec Radford, who stood up for the guy who built the original GPT one model. What that happened was that who had the intuition from seeing a lot of AI systems that have been trained that ailing the system further would be very interesting. And so that conviction, which didn't bear fruit immediately led to GPG to GPG was an interesting system. But it seems like it was on track to be amazing, from where we stand now. But actually, when GBD two was released, Google released a paper I think, within a year, calling it sort of antiquated to fudge to algae, I said that these language models weren't really going anywhere. They had better things. And Dario really had my conviction that was bolstered by the research that many of us did that expose these exponential trends that just was even a CPG. One wasn't commercially valuable even if GPT two was impressive, but not ready yet. But we work on some trend scaling towards towards much more useful, powerful systems and and so this is sort of the real world evidence for these trends. So if you look at the amount of computation that was being used, and still is being used in the AI field, it's really really growing exponentially. And an interesting thing that I observed by the and I'm not a chemist by but but I took chemistry in high school, and you might remember there's this number called Avogadro's number that just said, like six times 10 to the 23. It's like a billion billion billion. And I always thought, well, chemistry is really weird. We have this giant number and these numbers are not useful in any other place. And now we're at a point after sort of 70 years of computer science, where we can train models with more than a Avogadro's number of floating point operations. So these models like sort of GPT, three GB four, etc. are trained using something like 10 to 24 floating point operations. It's a tremendous tremendous amount of computations sort of human civilization was able to put together to train systems and this is the track, but I think everyone in this audience probably noticed that this is really expensive. Training these larger, larger systems cost more and more and more money on even with with advances in computer chips. So why did Dario and like we continue to have to commit. This is where things are heading even though these systems are so that comes from scaling laws, which are sort of precise scientific discovery we made a few years ago. First with language models literally with GPT one and GPT two, and then we investigated generative AI across the spectrum of modes, multimodal systems systems asked to solve math problems systems asked to generate images, all the same way. There's this sort of grand unification that occurred in AI where we can use the same type of systems with different kinds of data and get strong results regardless. And so what all of these rainbow plots show is that there's this increasing trend where models are becoming more and more capable, they're fitting their data better and better and better as you scale up the amount of compute data and model size that used to train them. So based on these scaling trends, and you can see on these plots range over sort of a factor of a billion in steel from tiny amounts of computation, vascularized truncation, because of these scaling laws, we thought, we'll make a bet we'll train a model or maybe worth $10 million and see what it can do. And that's that's what led us to build d3. And that's I think, what's what's sort of powering generative AI these days and so there's sort of look at the explosion of AI capabilities for access for time, there was sort of this early era of AI systems that were trained to do very specific prescriptive tasks like playing a single board. Game, and early researchers would try to train the system to maybe generalize to other kinds of games. Can you do play Space Invaders that also play Pong, but the results weren't very good. All these systems were trained to do sort of one thing to classify images, etc. And then around 2018 2019, we started to get these generative AI systems deep early language models that are able to actually do more general type of types of tasks and GPU IGB hmm. And then in the last couple of years, these systems have finally become sort of useful for something. And firstly, they were they were useful for, say, autocomplete for code. They were pretty good at chatbots. And then, very, very rapidly. Last couple of years. We've got systems that are not very smart and not very general to systems that can do as well as high school students. Sign most. Maybe as most college students they can pray. They can exhibit complex behaviors like adapting themselves to users in an undesirable way. We all thought that could be systems can improve themselves by thinking out loud by printing themselves. The system can play complex games involving perception. So this has really blossomed and I think the sort of key the key drivers here are scaling up compute and also, as more and more researchers get excited about this field, there has been a really rapid algorithmic flash virus that makes these systems more efficient and makes it even better as an investment to state of the art AI and really the opposite of viruses. It's faster to demonize now, that explains how these systems are becoming more general and more powerful. But why? Why do we need safety worldwide needs durability, etc. And, and this is something this is something that we we've all we've all encountered and even when we try and make these systems, or do we want things are moving so quickly that researchers make mistakes and and models that we deploy, make mistakes and I think these are sort of cherry pick examples. But they're these problems versus systems. I think the friction of saying things or they hallucinate and don't give you reliable, that sort of permission. And when we serve a customer as an object, the number one thing people would like to see in the group about these AI systems is to be more honest, to be more factually accurate, to not lose me to be able to be trusted. And that's that's, that's why we're so focused on these questions. And we are making rapid progress in this area. It's something a lot deeper than many others are very focused on. But with the extreme speed of development of the technology is so far. So this is sort of the timeline of some work from folks that are robic and others on controlling steering and better understanding these models. So I'm early on in ancient days in 2016, there was this paper by Dario and others on
10:01
trying to make concrete the problem how are we going to control AI systems? And then there was sort of this steady progress, reinforcement learning for human preferences. was first developed about five years ago. That's what's heavily used in a lot of projects. There was simultaneously progress in interpreting how we work feature information and understanding the neural circuits in neural networks that the classmates gave in one way or another. There are these developments, scaling laws, they mentioned that explain why increasing investments are worth while and wider and more recently, we have quad and constitutional AI. So what is unusually active this is, this is a single slide and I have a complex subject, but to explain how you play works, I'll go back to how does how do these human in the loop or human preferences to control as so the basic way we do that is very simple. We have these generative AI models. You can sample from them and they'll say, Hey, do we have these systems? Sample two possible responses. And you have humans evaluate which response is that? And then you just train systems to reinforce the behaviors that are desirable and avoid the behaviors that are undesirable. So it's simply we have a pair of responses from models and we go into ratchet direction of good behavior, we disincentivize bad behavior, and with 10s, hundreds of 1000s of human conversations we can train our system. So what is constitutional AI and how does it differ? From that process? With constitutional AI, we can make the principles the AI system follow. Transparent, we can share them with the world. We can iterate much, much more quickly because we don't need humans in the loop at all. And we can work to improve future AI systems at at sort of a faster, faster. So the idea potentially is is really really simple. Once you understand this idea of human in the loop preference, trade instead of asking people say sub workers, what behavior from an AI Do you prefer? We ask the AI itself, according to a list of principles, this constitution, what behavior would be better and so we replace human with training human feedback with AI self evaluation based on the constitutional principles and so that allows the AI system to effectively train itself which means that you can you can iterate you can train a new AI with a new cluster of principles in a day rather than having to wait months to talk to foreign workers. Tell them what to do and like for more. And furthermore, it works as well or better for steering the AI systems away from our tendencies as as as you can leave them so this plot from one of our scientific papers all it's really showing is that there's this purrito improvement in how helpful and our list is defensive and non toxic racism, etc. The model is there's normally a trade off where if you tell AI systems don't help people rather vague, they become less comfortable generally and constitutionally add helps us to do. So, that is how we train fly we train fly with a mixture of constitutional AI and human feedback for from from from domain experts in order to improve its utility, say for software engineers or lawyers or other other potential customers because of course experts know best what they actually want to do is one of those things together. And we're finding a number of use cases we just we just started testing blood with customers back in January and February we just watched it we already have a lot of different use cases. So of course, flight can be used for search or content generation or summarization activity. It can be used as employee assisted. There are all different use cases which just sort of are kind of ignored in general, and they're gonna reach more, more customers and really quickly ensure privacy and safety of customer data. We're partnering with Amazon on Amazon bedrock to deploy why you could so I'll show you a couple of use cases and a couple of them a couple of videos. So if you use the go share, notion is integrated AI. notion was a very early mover in integrating AI into their platform, and there's all sorts of different features that clause means for within just to write to summarize, etc. So this is just this example. And this is a really exciting partner for us because they iterate really, really quickly. Whenever people on my team are frustrated by and can't do something, I tell them, Well, don't tell customers you can't do that because the two months and we'll be able to do that. And notions moving really really fast as well. So it's really exciting to direct them to add more features and I hope there will be another example of like maybe earliest customer is probably Hi, who has the legal workflow to make lawyers much more productive. This is obviously an area where there's a lot of value lawyers time is valuable. And so if you can make them more productive, that's helpful. Actually, we've had pushback from lawyers that they don't love this technology anyway, because they don't like our ad they're afraid that they literally asked us if it's only gonna take us 10 minutes to write this contract. Like what do we tell our customers what's gonna happen, but that's that's the problem for lawyers. So, rather, the AI uses blood to basically instantaneously edit long contracts to make them more favorable to one party or the other. So if you I mean, if you if you work with lawyers, you know that there's a lot of details in legal language that you could change to make things safer for you in court. And this is sort of an example of plot following
17:09
around as instructions to that party you're advocating for and so this is a video going by fast, but this is a video of a new feature that we just released and I think many people are interested.
17:42
So something something you make if you've been paying close attention to generate AI and generative AI language models is that language models have context window. Unlike us, they offer a number forever. They only remember for some finite number of words and they've recently seen and the earliest language models maybe only remember a few 100 words. That was that was the limit. So that number is expanded over time. The early version was applied at about a 6000 word or 9000 token context and we felt like many customers were getting support problems if you'd like to be able to get flooded a book or a giant document or or your financial report and haven't had to answer questions or my summary. And so we've with our newest release and plot introduce them we've introduced 100,000 token contracts, so that basically you can fit a document the size of the Great Gatsby entirely enclosed project. So in this video showing is the quad doesn't know anything about lampshade maybe most of you in the audience do like chase this great, great platform for combining different language models together to do complex tasks. Well, it doesn't know about like chain but you can plot the entirety of landscapes documentation, fly, pin and rank and explain code to use like chain with with once you can get a visa tire tire manuals by using new API. And so the final video is just about speed. So one thing that we've tried to do in making flight as useful as possible is reduce labels. No one really likes weight. for language models to generate samples were by word. And, and. But there are a lot of language models out there and a lot of horrible characteristics. But what is good samples about almost 500 characters a second, so that really for a lot of use cases where latency is important where you need to get an answer quickly. This is something that can be really, really optimized. We hope it's useful for you
