Okay, hi everyone. My name is CH. Uh, today I want to introduce you to this optimization question about how to optimize agentic systems. Uh, I'm the founder of the AutoGen project and recently renamed that to uh, into a new chapter called AG2, and I also joined

Google DeepMind recently. Previously I worked at Microsoft Research. So um, I want to uh, first introduce a little bit about our, like, motivation when we build the AutoGen framework. It's an open source program framework for for agents. Uh, so earlier, uh, in like 2022, uh, when we

saw the power of like GPT models, uh, and we we started wondering, uh, in this new generative AI era, uh, besides generating high quality content like text and images, where where is this future applications going? What's the trend? And how do we maximize these powerful models

to uh, make them execute increasingly complex tasks for the future? And uh, uh, around the year, around the time when GPT-4 was released, we made the technical bet that the agent will be the future and uh, for for multiple reasons. Not only can they become the natural interface uh, for

humans to interact with the digital world, but also they that will enable the AI to have stronger capability than just using a simple model. And the third reason that not many people are talking about is that agent programming can build a new, like a software architecture to in

us to build software in a completely different way. And uh, uh, we we made that bet very early, um, when many people still had the doubts about whether agent is a viable notion, um, and uh, but but we believe that's the future and it will take time for

people to realize it what we didn't expect is it took a much shorter time than we uh planned uh so uh like starting from uh later last year and this year uh we have seen a lot of interesting uh work uh applications built by developers in all different

domains and here's one example uh built by Emergence AI, a startup company based in here uh and they built uh this web automation agent that can help users uh automate tasks like booking flight tickets uh or filling out forms automatically from my email and so on uh they uh build

uh agents using this AutoGen technique uh which contains a team of agents uh in a hierarchical way and uh without using multimodal models just using the text-based language models uh but uh thanks to the power of this hierarchical planning and decomposing the task into smaller

pieces the agent has a very nice accuracy in understanding the HDO content and achieve the best performance on this Web Voyager Benchmark um and uh they have a lot of interesting lessons learned from building such a web agent application and if you look at the performance numbers uh it's

roughly about 70% overall on the different tasks so there's still a large room for improvement of this one uh way to improve it is to introduce multimodal models to read the text uh but there are in general big questions for ML developers

is how do we really optimize agents to meet the success criteria for uh production use cases and uh and that's the topic I want to discuss today um so in terms of optimization uh there are many different uh metrics that we want to optimize for uh first uh and

foremost important is really the quality so we first want agents to be able to meet a high quality bar before we worry about other factors and then you may want to optimize for uh the cost latency and even the manual effort of doing such optimization and how do we optimize for

these metrics there are also a lot of optimization choices when building such uh applications uh there are like the models to use like we need to choose which model which version of the model and which from which model provider to use what's the prompt to use and uh

there are many other configurations such as the tools uh or the human input uh how do we combine these models tools and humans in the loop and orchestrate them all to perform effectively um so in general uh to answer this kind of questions uh we need

to basically have two basic questions to answer so the first uh first question is uh how do we uh represent this large design space uh and uh make developers quickly navigate along along them and find the optimal point inside that and the second question is

how can we automate that process so this is a similar lesson we learned previously from uh the Auto autoML work automated machine learning work uh initially people build uh like standard machine learning frameworks like scikit-learn and um uh pytorch uh to enable us to

quickly uh try out different uh designs uh different design points in a large design space and then people start working on auto techniques to automate uh these efforts uh so uh when we started building Auto-GPT we have a similar journey so uh we I've been working on

This Flamel, the AL ml open source library, started from like about five years ago. So uh when we started building AutoGen, we also wanted to eventually get to the point where uh we can just take a task from the user and automatically present them with the optimal design of

an agent and agent systems. Uh but before that, we first need to create this standard interface and standard framework for developers to quickly build up all these agentic systems in the easy way. Uh so uh after some effort, we uh published the research paper earlier in August and then we

quickly uh got a big interest uh in in on on GitHub and received multiple uh WS uh but what excited me most is to see how quickly the community is adopting this work and started building all sorts of interesting applications uh from both startup

companies and large enterprise companies. Uh the agent uh agent e is one example in the web uh web domain. There's uh there are other use cases such as science discovery. Uh we have found uh a uh multiple scientific use cases for, for example, for protein design, for material

design, and even proposing new scientific hypotheses for for any arbitrary scientific domain using uh ontology knowledge knowledge graphs. And another example is the software development and data analytics. So um uh we have seen a lot of interesting use cases uh for from from the community. And then uh to summarize

these practices uh we offered a course on this uh de. AI called uh agent AI design patterns using AutoGen to talk about how we handled these uh uh different uh important fundamental foundational design patterns. Um but if we if we um just try to simplify this

The concept here, uh, the the mostly essential concept in AutoGen is quite simple. It asks the developers to think about arbitrarily complex applications in two simple steps. The first step is to define agents, and the second step is to get them to talk. So um, so uh this agent

notion in AutoGen unifies many different types of entities, including the language models, including the tools and human inputs. An agent can uh use uh any of these uh primitive capabilities to reply, send messages to other agents, and then receive requests from other agents, and then start performing actions.

You can use a mixture of these backends as well. And uh, how do we get uh from this relatively simple agents to more powerful agents that can perform more complex tasks? The answer is that we can uh compose this uh multi-agent chat in different ways, uh in different

conversation patterns. And uh, and then uh we can, for example, nest uh multiple agent chat inside of one single agent and make that agent uh be perform much more complex tasks. And you can recursively build this up and uh uh transition from the multiple agent to single agent, uh the

examples of multi conversations are like two simple two-agent chat, a sequential chat that consists of a sequence of predefined chats, and only the previous chats finish the next chat will happen. And they will carry over from the previous chats the most essential information to the next chats. There's uh

also the group chat, which is more flexible, so like, you know, we put uh everyone in the same group and they can speak in different orders. Uh, and this order is dynamically decided by the group chat manager who performs this orchestration. And you can also add

constraints to specify when to transition from one agent to another and by using the building blocks, uh you can easily, quickly navigate the large design space. We've seen a lot of interesting, innovative experimentation out of this using these building blocks. So now we're entering the next phase, uh of more automatic techniques to optimize for these metrics, uh from different dimensions. Uh here are at least a few examples of the agentic optimization techniques they target four different m

the agent optimizer for example targets for the quality and in the same time try to reduce the manual effort for optimization. Uh and the optimizing choice it tries to optimize is the tools uh that we can provide to the language model agents. This uh we can allow

the developers to provide a set of tasks and the agent measurer will automatically go through the tasks, try to suggest the tools uh to the agent and try them uh and reflect on how effective they are at the end. And then we can iterate this to refine the tools or create new tools

as we need. Uh and there are other examples like Captain agent who try to optimize for both agent to and otion and reduce manual effort. I will talk about that in the next few slides. Uh and there are a few other examples trying to optimize for cost and quality or latency. I think

one um meta point I want to make here is that by unifying the different types of entities using an agent concept and by unifying the complex, uh the complex application uh combining all these different types of components using a simple multi-agent conversation paradigm, uh it makes it much easier to reason

reason about these applications and also makes it easy to perform optimization in a relatively simpler, high-level interface. So this is similar to how we lift up the abstraction of programming from Assembly Language to functional programming to object-oriented programming, and then uh now we have the chance to uh embrace a brand new

paradigm of agent-oriented programming. So I'll dive into the captain agent uh idea in the next few slides, and and and this is also a promising paradigm that we can eventually potentially integrate all the different op TS into one single piece of

system. So the idea is that we want to save the developers the manual effort of manually creating these multi-agent systems. Uh so uh we will try to provide a functionality called autobuild so that uh based on the description, we can automatically generate this a right

agent team to solve the task. And primarily there are two uh different paradigms: uh static build and dynamic build. In the static build, we will take the task description and generate the agent team at once, and in the future, we use all the same team again and again for

future tasks. The adaptive build is uh more flexible; it uh will uh take the task description, uh try to decompose the task into multiple sub-subtasks, uh and uh for each subtask it tries to create a different team uh based on the requirement, uh so that the team can be

used for this particular step, and in the next step we will we can replan uh and and to create a different team so that has a uh enable us to uh address uh the different task requirements and also uh handle the more complex task to uh

divide and conquer them so here's one example of how cabinet agent works uh it takes the initial problem requirement uh the user when the user talks to this cabinet agent it uh it has the impression that it's talking to only a single agent but under the hood the cabinet agent can uh

leverage a library of uh multiple agents of different roles uh and select them dynamically and create this message group chat for them to solve the task and after each uh conversation finishes the cabinet agent will have a reflection on like what uh was solved properly for

this particular step and what is uh still needed to improve uh and then in the next step it has a chance to fix the problem and here's a like very concrete example about how to answer a question that needs to analyze the YouTube video uh and understand content and answer the

complex question um and you can see that the agents some of the agents will be able to generate code and others able to execute code and there are other agents to review the process and so on uh and it achieves a quite strong performance compared to all the previous

uh work with similar flavor and also if you compare the static build and the uh dynamic build uh there's a big jump in the performance in many different tasks like solving math problems uh programming tasks uh data analytics and so on so I hope that gives you an idea about

uh some examples of the potential optimization techniques we can potentially enable with this uh uh foundational agentic programming framework and there are a lot more research we can do uh for example we can combine all the different techniques that optimize for different perspectives

and offer a system that automatically, uh, according to the matrix, different metrics to stress the optimal design point and, uh, and the dev can, uh, start from there to further refine them. Uh, I want to take a moment to acknowledge all the, like, open-source contributors. Now we

have this open governance, uh, new organization called AG2. We welcome everyone to join forces and develop the future of Agents together. Uh, thank you very much. Feel free to reach out on on Discord and platforms to for for [Applause] questions. Um, do I still have questions

for, uh, do I still have time for questions? Yeah, uh, okay, good. Uh, any any questions? Yeah, please. Yeah. Um, thanks for the great presentation. Yeah, right. So, uh, I have one question about, uh, the, uh, yeah, uh, um, about the measurement of the performance. Yeah, right. So, you, so you like in one of

the papers you said, like, it was around 85%. So, so my question is about like the failure cases and the failure cases can be simple. For example, if I, uh, asked to book a flight, I might just fail to book a flight. Yeah, or it could be like

booked the wrong flight, or it could be did something very weird, like, like RM minus RF, uh, slash, right? So, so how do you account for that in this, uh, you evaluation framework? Yeah, that's a very good question. Uh, one thing I want to mention is that, uh, using this, uh, agents, we can

potentially build automatic evaluation systems that help you, us, better understand how good my application is, uh, and, uh, go beyond the simple success rate because sometimes you can't tell the difference just by looking at the numbers if they're very close, but the tool can, can potentially make very

different kind of mistakes or failures uh this uh we have some research recently called Agent eval uh that's about uh creating several agents that automatically look at execution logs and suggest the different success uh criteria different dimensions to consider for this application exactly like what you mentioned if the

application is about flag booking the agent can automatically suggest several different sub criteria to check and then automatically provide scores for this particular uh run how good it is uh at each dimension and for the next run how good it is so that it can give an idea not

about the overall um numbers about different techniques but also the area that need further Improvement or the area that's already doing well so that provide more insights and you can further include human in the loop to provide more guidance to the agents so that they can refine their criteria

or or scoring uh part of it um so this multi-agent uh design can enable you have created agents that uh be responsible and focus on different success criteria and in some of those cases uh uh we see from the community they can generate a really nice um uh

report uh all the way from a single simple question can generate long and comprehensive report that are good at all the different criteria and that's because they have a different uh agent teams that each of them focus on different uh uh perspectives of them and eventually combine the result to to

generate a nice comparison report yeah okay uh one more question please yeah pass this one down hello thanks um as as these like um agentic frags are keep evolving and evolving and growing and there are so many durations along the way how do you mitigate um the

randomness in each situation and how do you would you suggest to like when the conversation starts and it's becoming like like a huge conversation and we can manage that randomness in it in in each conversation like in each uh interaction so uh I think that generally

there's the trade-off between uh the controllability uh and uh the adaptability to address new use cases that we haven't thought about You can define some very precisely controlled flows that um that reduce randomness but on the other hand uh they may be confined

to uh perform limited functions and cannot uh handle the new new situations uh we uh in AG2 currently we offer different conversation patterns uh that you can pick and uh according to this needs but in future we plan to also add some automatic mode so that we can uh

probably potentially uh suggest uh uh these different patterns automatically according to uh the the goal here yeah

Optimize Agentic AI Systems

Arize AI is the watcher, troubleshooter and the guardrail on deployed AI.

