all of you uh know Andrew Ng as a famous uh computer science professor at Stanford. He was really early on in the development of neural networks with GPUs. Of course, a creator of Coursera and popular courses like deeplearning.ai. Also the founder and creator uh and early lead of Google Brain. But one thing I've always wanted to ask you before I hand it over, Andrew, while you're on stage, uh is a question I think would be relevant to the whole audience. 10 years ago on problem set number two of CS229, you gave me a B, and I was wondering, I looked it over, I was wondering what you saw that I did incorrectly. So anyway, Andrew, thank you, Hansen, um looking forward to sharing with all of you what I'm seeing with AI agents, which I think is the exciting trend that I think everyone building in AI should pay attention to. And then also excited about all all the other uh on Sak presentations. So hey, agents, you know today the way most of us use Lish models.

is like this with a non-agentic workflow where you type a prompt and generates an answer, and that's a bit like if you ask a person to write an essay on a topic. And I say, please sit down to the keyboard and just type the essay from start to finish without ever using backspace. Um, and despite how hard this is, L's do it remarkably well. In contrast with an agentic workflow, this is what it may look like: have an AI, have an LM, say write an essay outline. Do you need to do any web research? If so, let's do that. Then write the first draft and then read your own first draft and think about what parts need revision and then revise your draft and you go on and on. And so this workflow is much more iterative where you may have the L do some thinking, um, and then revise this article and then do some more thinking and iterate this through a number of times. And what not many people appreciate is this delivers remarkably better results. Um, I've actually been

really surprised myself working these agent workflows. How well they work. I's do one case study at my team analyzed some data, uh, using a coding Benchmark called the human eval Benchmark released by open a few years ago, um, but this says coding problems like given the nonent list of integers return the sum of all the all elements are an even positions, and it turns out the answer is you code snipper like that. So today lot of us will use zero shot prompting, meaning we tell the AI write the code and have it run on the first spot like who codes like that no human codes like that just type out the code and run it maybe you do I can't do that um so it turns out that if you use GPT 3.5, uh, zero shot prompting it gets it 48% right, uh, gp4 way better 607 7% right but if you take an agentic workflow and wrap it around GPT 3.5, I say it actually does better than even gbd4, um, and if you were to wrap this

type of workflow around gb4 you know it it it also um does very well and you notice that gbd 3.5 with an agentic workflow actually outperforms gp4 um and I think this has and this means that this has signant consequences fighting how we all approach building applications so agents is the ter of around a lot there's a lot of consultant reports talk about agents the future of AI blah blah blah I want to be a bit concrete and share of you um the broad design patterns I'm seeing in agents it's a very messy chaotic space tons of research tons of Open Source there's a lot going on but I try to categorize um bit more concretely what's going on agents reflection is a tool that I think many of us should just use it just works uh to use I think it's more widely appreciated but actually works pretty well I think of these as pretty robust technology when I use them I can you know almost always get them to work well

um planning and multi-agent collaboration. I think is more emerging when I use them. Sometimes my mind is blown for how well they work, but at least at this moment in time, I don't feel like I can always get them to work Rel Lively. So let me walk through these four design patterns in the few slides, and if some of you go back and yourself will ask your engineers to use these, I think you get a productivity boost quite quickly. So reflection, here's an example. Let's say ask a system, Please write code for me for a given task. Then we have a coder agent, just an LM, that you prompt to write code. To say you def du task, write a function like that. Um, an example of self-reflection would be if you then prompt the LM with something like this, Here's code intended for a toas, and just give it back the exact same code that they just generated, and then say, Check the code carefully for correctness, sound efficiency, good construction, CRI, just write prompt like that. It turns out the

same l that you prompted to write the code may be able to spot problems like this bug in line Five. May fix it by blah blah blah. And if you now take his own feedback and give it to it and reprompt it, it may come up with a version two of the code that could well work better than the first version. Not guaranteed, but it works, you know, often enough for this be wor trying for a lot of applications. Um, to foreshadow to use if you let it run unit test. If it fails a unit test, then he why do you fail the unit test, have that conversation and be able to figure out fail the unit test. So you should try changing something and come up with V3. By the way, for those of you that want to learn more about these Technologies, I'm very excited about them. For each of the four sections, I have a little recommended reading section at the bottom that, you know, hopefully gives more references. And again just the foreshadow multi-agent systems I've described as a single coder agent that

you prompt to have it you know have this conversation with itself. Um, one Natural Evolution of this idea is instead of a single code agent, you can have two agents where one is a coder agent and the second is a Critic agent, and these could be the same base LM model, but that you prompt in different ways where you say one, your expert coder, right code, the other one, say your expert code review to review this code. And this type of workflow is actually pretty easy to implement. I think it's such a very general purpose technology for a lot of workflows. This would give you a significant boost in the performance of LMS. Um, the second design pattern is to use many of where already have seen, you know, LM-based systems, uh, using tools. On the left is a screenshot from, um, co-pilot. On the right is something that I kind of extracted from, uh, gp4, but you know, LM today, if you ask it what's the best coffee maker, web search for some problems, um, will generate code and run code, um, and it turns out that

are a lot of different tools that many different people are using for analysis for gathering information for taking action for personal productivity. Um, it turns out a lot of the early work in two use turned out to be in the computer vision community because before large language models (LM's), you know, they couldn't do anything with images. So the only option was that the LM generate a function called that could manipulate an image like generate an image or do object detection or whatever. So if you actually look at literature, it's been interesting how much of the work, um, in two years seems like it originated from vision because LMs would blind to images before, you know, GPT-4 and and and Lava and so on. Um, so that's two use and it expands what an LM can do, um, and then planning, you know, for those of you that have not yet played a lot with planning algorithms. I I feel like a lot of people talk about the chat GPT moment where you're wow, never seen anything like this. I think if not used planning alums many.

people will have a kind of a AI agent wow. I couldn't imagine the AI agent doing this. I've run live demos where something failed and the AI agent rerouted around the failures. I've actually had quite a few of those moment wow. You can't believe my AI system just did that autonomously but um. One example that I adapted from a hugging GPT paper. You know you say this general image where the girls read where a girl is reading a book and it posts the same as a boy in the image example. Jpack and please subscribe the new image for your voice so give an example like this um. Today we have AI agents who can kind of decide. First thing I need to do is determine the post of the boy um. Then you know find the right model maybe on hugging face to extract the post. Then next need to find a post image model to synthesize a picture of a of a girl of as following the instructions. Then use image to text to and then finally use text of speech and today we

actually have agents that I don't want to say they work reliably. You know, they're kind of finicky. They don't always work, but when it works, it's actually pretty amazing. But with agentic loops, sometimes you can recover from earlier failures as well. So I find myself already using research agents for some of my work, where one piece of research, but I don't feel like, you know, Googling myself and spend a long time. I should send to the research agent, come back in a few minutes and see what it's come up with. And it sometimes works, sometimes doesn't, right, but that's already a part of my personal workflow. The final design pattern multi-Asian collaboration. This is one of those funny things, but, um, it works much better than you might think, um, but on the left is a screenshot from a paper called, um, chat Dev, um, which is completely open, which actually open source many of you saw the, you know, flashy social media announcements of demo of a Devon, um, chat Dev is open.

source it runs on my laptop. And what Chad Dev does is an example of a multi-agent system where you prompt one LM to sometimes act like the CEO of a software engine company, sometimes act designer, sometimes a product manager, sometimes a tester. And this flock of agents that you built by prompting an LM to tell them, You're now Co, you're now a software engineer. They collaborate, have an extended conversation so that if you tell it, Please develop a game, develop a GOI game, they'll actually spend, you know, a few minutes writing code, testing it, iterating, and then generate a like surprisingly complex programs. Doesn't always work. I've used it sometimes, it doesn't work, sometimes it's amazing, but this technology is really um getting better and and just one of design pattern. It turns out that multi-agent debate where you have different agents, you know, for example, could be have ch GPT and Gemini debate each other that actually results in better performance as well. So having multiple simulated air agents work together has been a powerful design pattern as well um so just to

summarize I think these are the patterns of seen and I think that if we were to use these patterns in our work, a lot of us can get a pretty boost quite quickly. I think that agentic reasoning design patterns are going to be important. This is my small slide. I expect that the set of AI could do will expand dramatically this year because of agentic workflows. One thing that it's actually difficult for people to get used to is when we prompt an LM, we want to response right away. In fact, a decade ago when I was having discussions at Google on what's called a big box search, we type a long prompt. One of the reasons I failed to push successfully for that was because when you do a web search, you want a response back in half a second. That's just human nature. We like that instant grab, instant feedback. But for a lot of the agent workflows, I think we'll need to learn

to dedicate the toss and AI agent and patiently wait minutes, maybe even hours, uh, for a response. But just like I've seen a lot of novice managers delegate something to someone and then check in 5 minutes later, right, and that's not productive. Um, I think we need to, it be difficult, we need to do that with some of our AI agents as well. I saw, I heard some loss, um, and then one other important trend: fast token generation is important because with these agented workflows, we're iterating over and over. So the LM is generating tokens for the elm to read, so be able to generate tokens way faster than any human to read is fantastic. And I think that, um, generating more tokens really quickly from even a slightly lower quality LM might give good results compared to slower tokens from a better LM. Maybe it's a little bit controversial because it may let you go around this loop a lot more times, kind of like the results I showed with gbd3 and an agent architecture on the first slide, um, and

cand I'm really looking forward to Cloud 5 and uh CL 4 and gb5 and Gemini 2.0, and all these other wonderful models that may are building. And part of me feels like if you're looking forward to running your thing on gp5 zero shot, you know you may be able to get closer to that level performance on some applications than you might think with agenting reasoning. Um, but on an early model, I think I think this is an important Trend. Uh, uh, and honestly, the path to AGI feels like a journey rather than a destination, but I think this type of agent workflows could help us take a small step forward on this very long journey. Thank [Applause] you.

What's next for AI agentic workflows ft. Andrew Ng of AI Fund

We help the daring build legendary companies – from idea to IPO and beyond.

What's next for AI agentic workflows ft. Andrew Ng of AI Fund

What's next for AI agentic workflows ft. Andrew Ng of AI Fund

스크립트