hello everybody Adam LK here, and today we're going to intuitively break down the flows of six main approaches to building out a genetic architectures for large language model-based applications. If you're new to agents or unsure what sort of an agentic approach to building language model applications is, I have another video which will pop up right here right now that you can go check out. That gives a broad overview and a good example and breakdown as to what sort of agents are and what it means to sort of approach building LLM applications with an agent approach in mind. However, in this video, we're going to break down six main architectures that I've seen and they're Associated kind of papers and workflows intuitively. I think this fills a niche because a lot of the times these applications are very code heavy and not a lot of people truly understand sort of what's going on inside the actual execution of these architectures. So, I wanted to put together and break down both basic reflection and different abstractions of planning and execution so that you can better understand how

that's set up, and maybe even create an architecture of your own someday. So, without further ado, the first three approaches are really going to be focused around reflection. So, to break down what kind of basic reflection entails, it's super simple and kind of follows this flow. Your initial user query is input, and then an initial response is generated with the first prompt. So, let's say that the application might be an essay writer. You might input a request for Generate me an essay on, you know, woodland birds or something like that. That initial response will then be generated, and then it goes to a reflection step. The reflection step would be an additional prompt, let's say an essay grader, something that almost acts like maybe a teacher grading an essay, and then its whole point is to generate a critique. So, the critiques are sort of recommendations for improvements and different reflections and thoughts on the initially generated context and initially generated response. This is then fed back into a generation stage where that generation stage will hopefully take those critiques, take all those thoughts, um, different improvements.

that could be made. That itself generated and then remake a second response. This repeats sort of and number of times until you're happy with that final output. And this is pretty much about as basic as a reflection Loop can get: just generating the response, reflecting on it, and generating critique, regenerating, and then doing that sort of an arbitrary number of times until you hopefully get sort of a better output back. This is as opposed to just generating an initial output and then using that first output. So let's see what that actually looks like when it runs. And so all of these different agentic approaches, what I've done is gone ahead and ask them sort of the same question and use the same model. The current input question that I was interested in is: what are the current trends in digital marketing for tech companies? And then the model that I used for every step is going to be the recent April 9th release of gp4 Turbo. And so one quick note is that not all of these agents are set up to sort of do the same

task have the same tools or even have the same prompts so this is not necessarily a good scientific comparison or determination of what the best architecture might be. However, it should give you a good idea of the different approaches and what that sort of looks like. So for the basic reflection, we see the time that it took was about 118 seconds and about 18,160 [Music] generate a review. And this review looks a little something like this: talks about topic relevance, introduction of new trends, well covers well-established trends, could benefit from the inclusion of emerging trends such as voice search optimization, etc., etc. And then looking over here, we can see that it generates, reflects, generates, reflects, generates, reflects three times before a final generation here at the end. And so then this final output is what we see back here in this nice format. So I just took the markdown that it output and then rendered it here. My general thoughts on this is that it took a lot of time, there's a lot of processing since it generates a decent amount of

text and then kind of considers and has all of this in its you know quote unquote memory each time that it goes through and reviews. So it's a little bit token heavy and a little bit um you know time intensive to actually read regenerate every step of the way and then regenerate a full report every step of the way. And then another thing here is that as you can tell there's no actual tools attached to this basic reflection agent and so it's making a lot of interesting claims. Things like you know video content and interactive media indicating that video will account of for 82% of all internet traffic by 2022. And you know Amazon's use of personalized marketing techniques as it reportedly increase their conversion rates by over 202%. I actually looked up a few of these things and it turns out that 99% of the actual numbers here are completely made up. It's nothing that it's actually outputting here is very grounded in any sort of Truth um since it doesn't have access to things like web search tools.

which a few of the later agent architectures that I go over will have. It's pretty much just generating to generate, and a lot of its internal sort of waiting does not accurately get a lot of these um, a lot of these numbers right are very specific things. So that's certainly one thing to uh, one thing to consider when using sort of just basic reflection with no grounded tools. And just as my point here says, you know, it's making things up, and it's a little bit of slow processing, slow for processing, and the token usage is high. So however, overall super easy to implement basic reflection. Moving on to the second example, this is one that we have a little bit of actual research on. This is the architecture proposed by the paper Reflection Language Agents with Verbal Reinforcement Learning. Not 100% sure if it's reflection, reflection, or what exactly it is, but I'll just go with reflection because that's how I'm reading it. And it's based on this paper that came out in October of 2023 and takes this sort

of basic reflection idea and abstracts it a little bit further. And then finally introduces different things like tool use. And so the general flow that this takes is as such: Your user query is input. And then an initial response is generated along with the self-critique. And then also suggested tool queries. So for this quick example here, my example question was: Why is reflection useful in artificial intelligence? And then after a little bit of verification steps, and generating that first response, we get this nice initial response. And then it also will generate its own critique at the same time. So it says here that it's, you know, superfluous. Answer might be a bit too general. And its discussion or reflection without diving into the technical mechanisms or specific use cases provides good overview but lacks depth. And then the one tool that it's provided with here is a web search tool. And so it generates also some search queries: real-world examples of reflection in AI technologies, enabling reflection in AI, etc., etc. And so that's this initial response. And then what

differs is instead of actually having a reflection immediately, it takes those suggested tool executes and does just that. So the suggested tool queries are executed. In this example, web searches for more information, and then the original response reflection an additional context from the executed tools are sent through a revision prompt. So that's where this revisor here, which is going to be another language model prompt, is going to tackle this. So once that initial response is made, the tools and other context are executed and put in, and then the reviser is what gets all of that information all at once to generate a revised response. So the revised response will then update the answer, create a new self-reflection, and then new tool queries. And so here we see that this revised answer is here. It's also citing some things in here with actual references that it pulled from the tool Search tool, and then it sort of loops this way because it's made more search queries. So it'll get additional and additional context from the web using this tool, and it'll repeat this.

step and number of times until it's happy or until we hit sort of our arbitrary limit of what we want. At which the end result will be sent to the user. The biggest difference here between basic reflection and sort of this reflexion actor is that the actual tool execute step is separate. So instead of having everything just be regenerated all at once, it will take sort of those suggested queries, execute them all, and then all of the context along with all of the prior um context required, including the generated response, is then looped through and follows this until the end. Hopping down here to the testing area, then we prompted the reflexion actor with what are the current trends in digital marketing for tech companies? And it took only about 69 seconds and 24,000 tokens to generate this response. So hopping over to Langs Smith to see if we followed this precisely, let's check it out. And so yeah, what we're seeing here is the input current trends in digital marketing yada yada, and then the output is this first generated response.

and then also the search queries and a bit of a reflection. So we see this response actually be generated here in this drafting stage. And then these search queries initially are executed here in the execute tools stage. So this is exactly what we were seeing here with the initial response and then tool execution. So all of these tools are executed. And then the reviser over here, if we hop down here, is able to take that see all of the context from the web search here and then actually revise and answer. And then this is where it enters a loop of reflection. So you can see that it is specifically referencing missing things and also specifically referencing superfluous things. And then now that it's using a um web search tool, it's keeping track of references. And with that, it loops, executes the tools, revises, executes the tools twice more. So this is another example of kind of running through this about three times, similar to what we did with the basic reflection agent until finally here at the end, we have the final all the way down at the bottom.

here we have the final revised answer with still some reflection, but we take that out and just end up with our final report. Here, it actually took less time and less processing time than the general reflection agent. And then if we are to actually compare these tokens, the big difference here, I would say, is that this can vary from sort of the information that it's getting from the web. So, a lot of the times, the actual web scrape that it gets from its web search tool can be a lot of tokens. So, the cool part about it, though, however, is that we almost knocked the time down in half by taking out this execution stage and revision stage and having it loop like that. The third architecture, the more reflection-oriented approach, which is highly abstract, it's language agent research, or LATS for short, and it's based on this paper Language Agent Research Unifies Reasoning, Acting, and Planning in Language Models that came out December 5th of last year. And it's basically using large language models in three ways. It uses them as the actual

agents to do both the generation the value functions of the tree search, and then also as optimizers for for the tree with the general framework of a Monte Carlo tree search approach. And so in Len Chain's diagram here, they have this wonderful little synthesis of it where it kind of just repeats until solves. You know, select node, generate new candidates, act, reflect, and score, and then back propagate up through. But let's break that flow down a little bit more specifically, especially for those who aren't familiar with things like, you know, Monte Carlo tree search. Might be one wondering, what the hell does that mean? Doesn't matter, we're going to break it down real easily. So basically, First Step as always is the user query is input, that's that first generation, and then the initial response is Generation generated as a starting root node. And in this case, that root node can be either an answer or a tool execution. What we're seeing here in the generate initial candidate step right here is that the first route was actually just a tool.

search and so in this specific flow example I had you know generate a 500-word report on what Cisco Systems is doing with artificial intelligence. And then it says it wants to use my search results tool with the query Cisco Systems artificial intelligence initiatives. And so then that is actually executed. And after that, the reflection prompt generates a reflection on the output, whether it's the output of the actual tool executed or the generated response. And then generates a score for the output and then determines if a solution is found. So it's saying here then that the reflection is the search result, you know, provides valuable information on Cisco's initiatives with AI. The company has developed blah blah blah blah yada yada yada all that stuff. It gives it a score of eight and it says that this accurately found the solution. Then following that, an additional sort of n candidates are generated with the context of the prior output and reflection. And this sort of expands our tree out. So coming from this node and the reflection of it, we generate then in this sense one

two three four five additional candidates. And those candidates are then going to be sort of, if it was a tool use, it's going to be just plain generation. If it was generation, then it might be further generation or tool of use. And in this case, it looks like it was just generation. So coming from the context of the user query as well as the web search tool that was executed, it generated five different outputs of actual reports. Then on top of that, then it sort of repeats this, the reflection prompt grades scores and determines each new candidate. And then the scores for the best kind of trajectory are updated. So it'll pick the child node from that first initial generated node. And the one with the highest score is going to be the new candidate to start from. And starting from then, the best child node, more candidates are generated for the next step with the context of the prior output. And then the reflection and scoring is executed further and further and further. And this cycle repeats until either there's a high enough determined score or a Max search depth is reached.

so in short, basically, this diagram really confused me at first when I was trying to break it down. But in essence, all that this does is it will generate something initially, and then it'll generate sort of candidates from that executed initial node, and then it chooses the best one, and then from that best one generates another number of candidates for the next step, chooses the best one, and then generates another number of candidates until you either hit a max depth or, you know, iteration of nodes, or a score is actually reached. From that generated score of from the language model jumping down to the testing area, Lats actually was able to generate its report on digital marketing trends in only 30 seconds with about 8,000 tokens used. And this sounds phenomenal in comparison to a few of the other architectures that we've just went over. However, there's a lot of different concerns to be had here. So let's first hop over to the trace and see this in action. Great. And so looking at this, what

we can see is that our initial input is what are the current trends in digital marketing for tech companies. And then it does the generate initial candidate. The initial candidate was a tool call with current trends in digital marketing for tech companies in 2023 which are Tav. Search tool gives us here. It then reflects on this and grades it so. So yeah looking over here ah here we go. It reflects on it. The assistant provided you know comprehensive overview of current trends and digital marketing scores at a 10. And then says it found the solution. So you might be starting to see some of the issues here when using language models as every single part of your tree search algorithm. We then expand from this initial candidate with ah I see one two or no one two three four five different Generations. We generate this candidate here which has a report and then five different Reflections. And so all of these scores as you can see are starting to be around like nine. It's saying it found the solution. And since all of these are pretty high score and say that it found

the solution. One of them was chosen, and then finally, it was all output all the way at the end to this report here. So what you're probably seeing here is not great if you know anything about sort of different tree search optimizers and other things like that because what's going on here is that the actual scoring of the generated responses and the reflections of the responses, given that it is an LLM, is very indeterministic. So using an LLM as the actual sort of optimizer for this for the generated responses can be a little bit finicky. It tends to overscore, I would say, and it tends to be very ready to sort of accept an initial answer if it seems to be somewhat related to the input query. And as we can see, it didn't really generate too much further past sort of one initial expansion because it determined that everything had been found. So you could force this to generate more nodes, or some of the different approaches would be to use a better optimizer that isn't language

model-based and might be more, you know, hardcoded in and using that. It would be a lot less, um, lot less finicky and a lot more reliable for actually scoring the desired output that you're looking for. However, you know, was able to give me a nice graph-founded report on digital marketing Trends. But one of the other things is that it does have to generate a lot of text, especially if it's generating, you know, five or six candidate nodes per each, you know, generated node. That can really run away with token counts if you don't have sort of a Max search depth setup or if you're generating a lot of text all at once. So now that we've gone over some from basic to very abstract and complicated approaches of actual reflection, let's move on to sort of the last three here that are more based around a different kind of reflection, more of a planning and reasoning stage as opposed to direct reflection. So the first one is actually named plan and execute, and it is based on this paper plan and solve prompting improving zero.

shot Chain of Thought reasoning by large language models that came out around May last year. And it takes the reflection idea and breaks it down into a little bit of subtasks more like task decomposition than actual reflection, which can be sort of argued as a type of reflection as well. And so this one I find to be pretty interesting because the general flow of it is that your user query is input as it always is. And then instead of generating something right off the bat, an initial planning prompt comes up with a step-by-step approach to sort of completing the query. So for this objective input, which was, you know, What is the hometown of the current Australia Open winner? The plan here is then: identify the current year, find out who won the Australia Open in the current year, and then search for the hometown of the winner identified in step two. First step of the plan is then given to the agent, which either generates or executes a tool, which is the single task agent. So it'll look at that first step and then the original query, the original plan, and

the past step output are given to a replan prompt that either updates the plan or returns the output to the user. So once the first task is executed, it will give it to this replan. The replan takes a look and says, Have we finished the task? Do we need any additional tasks, or are we good to go? To respond to the user, if it is good to go, it responds to the user. If not, it updates the step. The updated step is given back to the agent to either execute or generate it. Updated generation with the context of the prior steps is then looped through. Input and plan are given back to the replanner, and this repeats kind of end times until the replanner determines that the answer is adequate and returns it to the user. So basically, a plan is made. The first step of the plan is executed, and then we take a look and reflect, see if that first step actually answers the entire plan or if we need more tasks or if we're good to go. If we need more tasks, an updated task

list is made, sent back to the agent that either executes or generates, and then you reflect again if it is good to go. The final response is sent over to the user. And so heading down to the agent testing environment here with the plan and execute approach, we saw that this took only 24 seconds and about 3,000 tokens. As mentioned before, tokens can vary wildly, especially with things like web search tools. However, I've seen that with a lot of these, the actual time tends to go down. And I think this is a bit more of an interesting approach than the actual reflection. The decomposition of it into different subtasks can be more easily executed upon from llms and breaking up sort of those complex problems into more simple steps are easier to sort of look at and actually see happen, especially with these language models. And so taking a peek at the trace here, what we can see is, here we go, the input were the current trends in digital marketing for tech companies. And then we have a planning step. So the planning step is research.

recent articles reports and studies on digital marketing transends for tech companies identify key trends mentioned in these sources such as AI integration personalization analyze how these Trends are being implemented summarize the findings to outline the current trends in digital marketing for tech companies so then we execute here here we go we execute the first step which is research recent articles reports and studies on digital marketing trends for tech companies and then what it does is it searches the web for recent articles reports on digital marketing transfer tech companies and then generates a bit of here we go an output here based on that and then oh it even gives some references here we shoved that generation then to the repl planner and the repl planner looks at you know the input the plan the past steps everything that's been made and then determines that based on the information provided appears that all necessary steps have been completed to identify and summarize blah blah blah blah blah no further steps are needed and this summary can now be used for analysis and Reporting so in one shot without any actual need

to replan it determines that was able to accurately answer the question that then gives us this output over here. And so a few notes that I had on this is that, you know, forcing a planning stage makes it a bit more efficient than a series of revisions, since it tends to sort of, you know, approach with a plan as opposed to just generating wildly and then generating a reflection, which it will always do if you always ask it to reflect on something. So it'll always generate something, but this tends to generate, you know, decent responses quicker with a little bit of its own guidance following a plan instead of explicit reflection. The next two architectures here then are going to be some pretty interesting optimizations of the whole plan and execute approach. And the first one that we'll look at here is reasoning without observation, which has a phenomenal name, which they called Riu and is based on the paper Riu decoupling reasoning from observations for efficient augmented language models that came out in May of last year.

short this really combines this multi-step planner stage with variable substitution for very effective tool use to reduce both token consumption and execution time with a full chain of tool use in a single step so let's break that down as always. We start with the user query as the input, and then the planner generates a plan in the form of a task list. This task list includes special variables as the arguments to allow for variable substitution. A good example of this is up here in the diagram. So the plan, let's say our question was who is playing in the next Super Bowl or something like that, or the upcoming Super Bowl, I should say probably, and the plan here is: I should look up the Super Bowl contenders. And then the first variable E1 would be search for the Super Bowl contenders, and then the plan would be get the first team, and the step E2, this variable E2 would be a large language model call for the first team from the results of E1. So what you're seeing here is that this full task list is generated with

variable dependencies. However, these variable dependencies are persistent throughout this entire task list so that it can be executed all in one stage. So, to just reiterate what that is one more time, looking at this plan: identify the winner of the 2023 Australia Open. Let's say the Google search would be George. And so then here in plan number two, find the hometown of the identified winner from E1. E2 would be Google hometown of, and then this E1 placeholder variable here, which is equal to this output, would be George. So then E2 would then become equal to Google hometown of George. And so then that would be E2, so on and so forth. And so then as each step is then executed, output is substituted into the placeholder variable of the next step and fed back into the LLM agent. Once all of the steps are complete, the plan and the evidence from the tool execution is fed to a solver prompt, which generates and returns a final result response to the user. So, in essence, what's happening here is instead

of executing one step of the task and then reflecting to see if another step should be done or more steps should be done. A full task list is made, and then the entire task list is executed, resolving all of these dependencies as each step is done. And then it's all. All of that context is sent to a solver, and then the solver creates the final output for the user. Hopping down then back to our testing environment, we can see that Riu took only about 21 seconds to do the same thing that the plan and execute stage did in 24 seconds. So let's take a look at the trace here. Seeing here, we see that the task is what are the current trends in digital marketing for tech companies, and then the plan that is created is, Here we go. E1 is Google current trends in digital marketing for tech companies in 2023. And then the second plan is, summarize the key trends identified, and so it's going to be an llm call for summarized key trends and digital marketing for tech companies.

from the results of E1. So, E1 is then executed with our Search tool. Current trends in digital marketing for tech companies. All this good text is then put into our second execution here, which is this summarization. And then the summarization is created and made into this report. And then that is all sent now finally into the solver, which has the task, the plan, strings, the steps, and then the results from each one. And then its output is this right here, phenomenal, which ended up being a little bit short but still good right here. And this, as we can see, is really an optimization of this plan and execute style agent. And well, it did have a little bit of a higher token count and as mentioned before, that could be due to the web search. It still took less processing time, so it's definitely a little bit quicker and a more optimized version of this plan and execute Loop. The final architecture example then here is a complete optimization of Riu called LLM compiler and it's based on the paper LLM compiler for parallel function.

calling from February 6th of 2024. And in short, LLM compiler pulls on the idea of directed acyclic graphs from compiler design to automatically and algorithmically generate an optimized orchestration for parallel function calling in sort of a React-style setting. What does that actually mean, however, is pretty simple. It's just a lot of uh, a lot of fancy jargon makes this sound a lot more complicated than it actually is, but let's break down the flow here. So as we've been doing, user query is taken as the input, and then there is a planner stage, and the planner stage takes in the input and generates a task list with placeholder variables for subsequent task dependencies, as well as thought lines for reasoning through the plan. So already starting off very similar to Riu with creating this sort of approach of creating a task list with dependencies. So looking here at this example, we have, you know, What's the temperature in San Francisco raised to the third power? And then the output is, We're going to do a web search with the current temperature.

in San Francisco. And then we're going to do a math problem, a math function call of the number one output raised to the third power, with the context being number one. And then this is joined. So then a task-fetching unit parses the plan and determines the interdependencies of each step. And then tasks that have no interdependencies are then fetched together and sent to an Executor to run in parallel. So that's where it sort of differs from Riu over here, where Riu will kind of go through and execute each one one by one, and then resolve the dependencies one by one if there are multiple steps that don't require both, um, that don't have any dependencies. Llm compiler and the Llm compiler approach will, um, execute those in parallel and then resolve everything at once. So looking at this second example here from the paper, the user input, you know, how much does Microsoft's market cap need to increase to exceed Apple's market cap, we can see that the first couple steps of the plan, the search for Microsoft market cap, and then

the search for Apple Market Cap. Don't have any interdependencies here. However, math here for number three, and then the actual LLM response for the fourth step here, do have dependencies on one, two, and three. So what it'll do is it'll actually fetch one and two, and then resolve them and execute upon them in parallel. So both of them happen at the same time, and then both of their results are then spit back at the same time. And once those results are spit back, the output of the executor is then, you know, fed back into the task fetching unit to resolve the dependencies, repeating back and forth until the plan is fully executed. The output of the fully executed plan is then sent to a joiner prompt, which either determines the final answer from the context or appends a thought to the end of the original plan and sends it back to the replan to add any additional steps if needed. A replan, a continuation of the plan, is created, and then it goes through and does the above loop until the joiner determines that

enough information is there to return an output to the user. So, this is getting a little bit complicated, but essentially it's taking the ideas from, you know, reasoning without observation and plan and execute, and combining them together with some algorithmic approaches for parallel function calling and parallel resolution of dependencies in the actual task. So, this is all to sort of optimize the speed and efficiency a little bit further. Great. So, hopping over to the trace here, we can see that the initial plan from the input What are the current trends in digital marketing for tech companies? was: first step, search for current trends in digital marketing for tech companies, and then send it to the Joiner end of plan. So, super simple plan makes a lot of sense when you see how this is all set up. It does the query for the internet, and then the Joiner prompt here takes all of this and then says that we've got enough to go on and creates a response after giving a bit of a thought around it. So, this response was then this.

final thing here that happened there so that pretty much brings us to the end of our very comprehensive overview of six main agent architectures from some popular papers. As always, all the code is going to be available via Lang chains platform. Full credits to them for setting up a lot of these good graphics and a lot of the code that I was actually able to use and learn from their YouTube tutorial series. Will be linked down below as well as all of the papers and as well as all of the Lang chain traces that we went over here in the video. If you enjoyed it, drop a like, hit subscribe, let me know if you have any questions in the comments, and I'll see you in the next. Thank you.

Breaking Down & Testing FIVE LLM Agent Architectures - (Reflexion, LATs, P&E, ReWOO, LLMCompiler)

Breaking Down & Testing FIVE LLM Agent Architectures - (Reflexion, LATs, P&E, ReWOO, LLMCompiler)

Scripts