Building a Full-Stack LLM Benchmarking Application with Pythagora

Pythagora enables users to effortlessly build a full-stack LLM benchmarking application in under two hours.

### Building a Full-Stack LLM Benchmarking Application with Pythagora
* 🚀 In this tutorial, I will demonstrate how to build a **full-stack LLM benchmarking application** in **under 2 hours** using Pythagora, with **1,600 lines of code**.
* 💻 The project begins with the installation of the **Pythagora extension** in VS Code and setting up the necessary code framework, including **Node.js**.
* 📝 I will create a benchmarking application that allows users to test various large language models (LLMs) against a set of predefined questions, emphasizing that this is a **legitimate and useful application**, not just a simple project.
* 🔑 After detailing the app's functionality, including **user authentication** and an **admin dashboard**, I will paste an extensive prompt into Pythagora to guide the development process.
* 🛠 Pythagora’s agents will take on different roles in the development team, with the **architect agent deciding on technologies like Node.js and MongoDB**, and the **spec writer checking the complexity of the prompt**.
* 🔍 As I interact with Pythagora, it will prompt me for necessary inputs, such as **API keys**, and assist with debugging while providing **real-time updates** on the code being written.
* 📊 Throughout the process, I will validate the app's functionality by running tests, ensuring **user registration works**, and confirming the **admin dashboard displays correctly**.
* 🔄 The iterative development cycle allows for **continuous improvements**, including adding features like **role management and pagination**, ultimately culminating in a **robust application that meets real-world needs**.

### Building a Full-Stack Benchmark Testing Application with Pythagora
* 🛠 The process began with testing the Open AI GPT-4 Turbo, where initial issues arose with **test names being deleted**, prompting a **bug investigation**.
* 🔍 As the team worked through the problem, they discovered that the test names were missing due to the sample data not including the **required name field**.
* ✅ After resolving the issue, the test creation form was updated to **prompt for a test name**, enhancing user experience.
* 📊 The team successfully implemented **real-time progress tracking** for test execution, allowing users to see the **status of their tests in real-time**.
* ⚙ Subsequent testing revealed a bug with the publish functionality, where an **error occurred despite the test being successfully published**, leading to further debugging.
* 📈 After several iterations and improvements, the application was fully functional, allowing users to **create, execute, and publish tests seamlessly**.
* 🌐 The final step involved **deploying the application live on the web**, showcasing its capabilities in benchmarking tests against different LLMs **without writing any code manually**.
* 🎉 In total, the project resulted in **1,637 lines of code** across various files, demonstrating the **powerful capabilities of Pythagora** in building **complex applications efficiently**.

watch me build a full-stack LLM benchmarking application, 1,600 lines of code in under 2 hours using an AI partner. Today, we're going to be using Pythagora to do all of this, and I've made multiple videos about Pythagora in the past, and you've always had great feedback about those videos, so we're doing it again. So, I'm very happy to be partnering with Pythagora once again on this video. So, let's just get into it. All right, so let's start building. So, we have the Pythagora extension installed into VS Code. I have the code framework I'm going to be using, which includes Node already installed. If you don't have those installed, go ahead and install them. And here is what Pythagora looks like. So, we're just going to dive right in. Let's click Create New App. The first thing it's going to ask for is a name. So, what is the project name? And today, what we're going to be building is something that I think I'm going to be

using all the time, which is a benchmarking application. And remember, when we're building with Pythagora, we're not building toy applications. This isn't like a to-do list or anything like that. This is a legitimate full-stack valuable application that doesn't take minutes to build. It might take hours, but it is something that we will actually use in our day-to-day lives. And so, what we're building is a benchmarking application, so something where I can give it a set of LLM test questions, and it will automatically test different large language models against those test questions, essentially a benchmark. And I want to use the benchmarks that I use in all my videos, so I'm going to be doing that. But this is going to be more generic than that. I can just decide to use those benchmark questions in this application. And I've already written up a pretty extensive prompt to start this project. I'll drop the prompt in the description below, and I'll just go over it briefly here, but it's pretty long, so not going to go through it completely. So

first the project name we'll just call it Benchmark. And here it says describe your app in as much detail as possible. And you're going to see how much detail we really describe it in. So I'm just going to go ahead and paste it in here. And all of this is describing how this benchmarking application will behave. So I'm just going to go through it very quickly. I want to create an app that will enable users to create tests for llms, run them, and publish the results. First, it needs a homepage with a welcome message, a list of published tests. Users should be able to click on a test that opens a test view page. It needs authentication, including administrators. It needs an admin dashboard, a page with a list of created tests, and a creation page for a new test. Here's the description about how I actually want the test created, more about what the llms will actually do with those tests, all the way down here, which large language models to use, and then finally a way to share it. So again, I'll drop this prompt in the

description below. You can also just click Start an example project if you don't know what you want to build yet and just want to try it out, or you can import an existing project. But since I already wrote this out, we're just going to click the little send button right here and get started. And what Pythagora, as I mentioned, is really good at is building full-stack real applications, valuable applications. And so the way that they do that is they have different agents that take on different roles as kind of a developer or development team that you manage. So we have the spec writer and the architect agent. So the spec writer is checking the complexity of the prompt. The architect agent is starting to plan it, asked me if I want to use the Node Express template. And since that is the framework I'm going to be using, I'm going to click yes. So the architect agent decides which technologies to use. So let's see, Node.js, MongoDB, Express, Express session, and so on, all the way down. So they're really choosing the full-stack application to

use architect agent made sure we have MongoDB installed. Here's some output. So imagine we were doing this through the terminal. This is exactly what we would be seeing there. Now Pythagora is asking us for input required on ENV. So ENV line 9. Please open the file. It does that automatically and it also highlights the line that we need to adjust. So what we need to do is insert your API key here. And that's what it's asking us to do. So I'm going to go grab that right now. All right. So I've grabbed first the OpenAI API key and I will be revoking this API key before publishing the video. Next I know it's going to ask for the Anthropic key. So I'm going to grab that as well. So both keys I'm going to revoke before publishing. There it is the Anthropic key. I forgot to remove this space right there. So let me go ahead and do that. I'll save. And now that I've gotten everything that it's asking I'm going to

go ahead and press continue. Okay, yep, it's asking for line 10. I've already done that, so I'll hit continue again. So, this is the interaction with Pythagora. It is basically doing all the work for me, and then when it needs me to intervene in any way, whether that's helping debug something or inputting custom API keys or anything else, it will ask me to help. It will tell me where to look to help and what it needs from me. And this is, if you remember from the last two videos about Pythagora, this is the interaction cycle with it. It's just very iterative, and it does a bunch of work, then it says, Hey, test this for me, and then if there are any issues, you describe the issues, and it walks you through debugging them. So, let's look at what's done so far. So, we have the tech lead agent creating the development plan, creating tasks, and it's putting together an entire development plan. It's not just going and building. So, developer agents starting new task with description.

update user model to include a role field with default value viewer. Do you want to execute the above task? So, you have three options, and I'll go over this the first time, but on subsequent times, I'm just going to go ahead and make the decision. So, do you want to execute the above task? You can either say yes, you can edit the task, and so you can give it more instructions or change how it is going to be done, or you can skip the task in case you don't want to do it at this time. So, right now, we're going to go ahead and press yes. So, the developer agent is thinking, writing code, and as it's writing new code, we're going to be able to see all the new code written right here. We're going to be able to see the diffs. It's really neat. And then also, you can see a progress bar right up here, so you can see the progress as you progress through building the applications. There's a logs tab. There are no logs at the moment, but any logs, front end or back end, will be

here you can also see the more detailed progress information in the progress tab. So you can see all the different tasks that were created, all the epics, and you're basically working through it step by step. Then the deployment tab, so we're not quite at that step yet, but we'll get there. Okay, now back to here. This is the tag that tells us we need to get involved, human test. So figuring out how to run the app, determining how to test the app, and now it's giving us instructions to go ahead and make sure everything's working so far. So action, start the server using npm start. Usually, you can do that through terminal, but it also has this nice little button with the rocket ship saying start app. You just click that, and it starts the app for us. All right, so we have the npm up and running. Just to make sure it is, we can click over the logs tab. There it is, server running at Local Host. So let's go ahead and copy that. I'm going to switch back to the development.

tab then I'm going to open up a browser. I'm going to type in Local Host, hit enter, and there it is, it's up and running. So now we've verified that that works. Let's switch back to VSS code, and we have three options at the bottom: everything works, make a change, or there is an issue. So far, there are no issues, so we can keep going. Next, it's saying we need to register, so let's test registering. So it actually wrote a lot of code already, and actually, you can see a lot of the code right here. Here's like off routes, for example, user.js. And again, as I mentioned, as you build out the application, you're going to see more and more code over here. And if we click the files tab, we will actually see a lot of these files being written as well. Next step two, follow the steps. We're going to grab this URL, very common auth register. Okay, so I just put an AM Burman and just a quick password. We're going to click register login, and it seems to

have worked there. There's the log out button, so we are logged in, so we verified that. Okay, so next step three, fill out the username and password. We've already done that, so it did work. Step four, open your MongoDB client or MongoDB Atlas and navigate to the database used by your application. Okay, so I've opened up MongoDB Compass to just check the database, make sure everything looks correct. Let's go back to make sure we know what we're looking for. So we're going to look for a user's collection and a newly created user document. It should have a username, a password that is hashed, and a role with a value viewer. So let's take a look. All right, so within Pythagora, we have users, and there it is. There's the user with the correct role, so everything is created properly. Let's keep going, so switch back. Everything works perfect. Yeah, so we've had no issues so far. This is great. Starting a new task with description create an admin user creation script that inserts the first admin user into the database. Do you want to execute? Yes. Okay, there so

it updated the code. Not only can we see the updates to the code, but we can actually see the if we were to click any of these buttons right here, but I'm not going to do that right now. So these are the different files that were just updated, and now it's asking for something from us. So input required on this line and create admin user right there highlighted already open. So input required provide the admin username and the password. Okay, so I came into EMV. I updated these two fields, and we should be good to go. So let's hit continue. It's asking me to do that for the password. I've already done on that, so let's hit continue again. And now it's asking for the database URL, but I already have that, so it's all good. Let's keep going. Okay, can I run the command npm run create admin with a 60-second timeout? Yes, you can. There it goes. Admin user created. Perfect. Determining how to test the app. Great. All right, now we got a bunch of tests that we need to run ourselves, so

action update the EnV with these admin credentials. We've already done that. Run the script to create the admin user using npm run create admin. In so that actually already ran, so I don't need to do that again. You could run it again, but you don't need to start the server. We already have one started. I go ahead and click it again. Open your browser. So next, we're going to log in. Let's start the app, and we're going to type in our login and password. And there we go. Now we're logged in, and we will go on to the next step. Okay, log in. It did work. Verified your login. Yep. Uh, open your browser to the admin dashboard. Let's do that. Page not found. Yep. Expected result. You should see an error page not found. Perfect. Uh, everything works correctly. Let's keep going. Starting new task. Implement an admin dashboard page. Yes, go ahead. Okay, once again, writing a bunch of code, and all of this code is going to show up on the left side over here, and not only is

it is writing the code, but it is explaining what it is doing, why it's writing the code as it goes, breaking down the task into steps. Code Monkey agent, now we're going to start seeing the code get updated over here. Yep, there it is, and everything's switching automatically to the code that is updating in real time. Okay, once again, we see all the diffs here, so lines created, lines removed, and of course, let me just show you when it's done. We can just click into one of them, and we can see the diffs just like that. Okay, testing again and again. This is just the iterative cycle that we go through. So, expected result, start the server using npm start. We could do that by clicking this button right there. Open the browser, log in with the admin credentials. Let's do that again, and there we go, and there's that new admin dashboard link at the top. Let's click it, and there it is, a very simple admin dashboard. It tells us the usernames and the role. Step three, login. Yep, step four, navigation bar admin.

dashboard yep and okay so everything works. Let's click it and keep going. Starting new task. So add functionality to the admin dashboard to allow the admin to change user roles between viewer and Creator. Yes, all right, right in code, and look up here. You can see we are progressing through the different steps of the application we're building, and this is one of 1, 2, 3, 4, 5, 6 different epics. And then within each epic, you can see all of the progress we're making. So here it is, Code Monkey agent working. All right, it's done, human test. We knew it was coming, determining how to test the app. All right, we got a lot of tests to do this time, so let's see. Start the server, done. Open the browser, log in. Let's just read through, and we can do it all at once in the browser. Visit admin, click the admin dashboard, refresh the admin dashboard. Oh, nope, sorry, one step back. We will change it from viewer to Creator, then refresh the dashboard, make sure things are changed, and let's stop there. So here

we are, and we can see the new functionality. There's the role. We're going to change the role of M Burman from viewer to Creator. We're going to click save. Roll updated successfully. Let's refresh the page, and we can see it has been updated successfully. Awesome. And now it says change it back to viewer. So let's go ahead and do that. Click save. Successfully saved. Reload. And there it is again. Okay, so now it says let's log out and then log into the non-admin user. Then it's saying let's test going to the admin page to make sure we do not have permissions. There it is. Access denied. Perfect. And then it's saying log out again and just make sure we can log in with the admin. So I'm going to log out, log back in with admin, admin dashboard. And there it is. So everything works perfectly. Let's keep going. Next, implement access control to ensure only admin users can access the admin dashboard. So we already did this one. Sometimes it'll suggest things we've already done, so that's okay. That's why

There is the skip task button right there, so we're just going to skip it. Next, create a new page for listing tests with a table displaying test names and creation dates. Yep, we definitely want that. Let's click yes. All right, so it wrote a bunch of new code. Now we test it again, so start the server. Easy click open to off login. Then we're going to log in with the valid credentials, and we are going to see the test link in the navigation bar. So let's do that. So I'm logging in as a regular user. There's the test tab. Click it, and no test available. So that worked. All right, everything works. Let's keep going. Implement a script to populate the database with sample test data. Yes, please. Okay, there it goes. All right, Code Monkey get to work. Populate test data.js. There it is, pops right open, and package.json reviewing. So we could see again all the diff right here. Can I run the command npm installed. NV with 60c timeout. Yes, you can. Can I run the command populate test. Yes, all.

right now back to testing. Start the server, click run the script to populate the database. We already did that. Open your browser, log in. We're already logged in basically. Look at the test page. There it is. Perfect. All right, that is definitely working. Everything works. Add a delete button. Yes, and so you get into this nice flow with your developer agent and your Pythagore team in general, and it just is super easy. So write in the code once again and let's test it again. All right, let's read through how it wants us to test it. Start the server, log in. Yep, log in using an admin account. Okay, visit the test page, click delete, confirm, refresh, make sure it's deleted, and let's do it. Okay, so I'm going to log out. I'm going to log in as the admin, go to test, and there are the delete buttons. So we're going to delete the code completion test. Let's click delete. Okay, and it's gone. I'm going to refresh the page just to make sure. Great, let's delete one more text summarization.

test okay gone refresh done all right easy peasy. Let's keep going all right let's do something else. So Implement pagination yes. I mean this is so cool cuz it's just so easy and now that I am in the flow of using it I can kind of almost do it faster. I can skip some steps. I know I don't necessarily need to log in every single time uh but I just want to keep an eye on what it's telling me to do and what it's telling me to test and by the way look how much code has already been written. All of these different files and so yeah this is so much more than just a simple Todo app. This is a real-world application that we are building from the ground up using pythagora. All right let's read through how to test it again. Open your browser visit login login with the admin credentials visit test. We're already there if there are more than 10 test scroll to the bottom of the page. The expected result pagination control should be there click on the page number.

verify that each page number displays the correct number of tests. Click through all the pages. Click on the delete button. Confirm it deletes all right. So basically just confirm everything again. Let's do it so back here going to refresh and it only has three tests. So I'm going to go ahead and run that script a couple more times just to get more than 10 tests in here. Back to Pythagora. Open up terminal. So we're going to run npm run populate test. Hit that once. Good. Let's do it again. And let's just do it one more time for good measure. So now we should have 15 additional tests in there. So let's switch back to the UI. Hit refresh. Okay. So it recreated all five but it didn't add five. So that's a problem. And let's just fix it. So I'm going to come back here. I'm going to say there is an issue. So I'm simply going to describe what the problem is. I'm trying to create more example tests but each time I run populate tests which is the npm script.

it recreates five rather than adding five. Can you make the script add five in addition to whatever examples already in there? Yeah, and the solution will be not deleting the existing test. Perfect, so it's rewriting that part of the code. All right, and let's run the script one more time. I'm going to start the app again, let's refresh the page, and there we go, we have a bunch of new tests. So that was so easy. I'm going to switch back and add 10 more, so that means we will run it twice, once, twice, and now because we have so many tests, we should see the pagination. All right, there it is, perfect. So yes, the issue is fixed. Back to the functionality we were adding, so I had to start the app again, refresh the page, page two, and that does seem to work perfect. Okay, now let's just try deleting one. Okay, page one, page two, and it always has 10 on the first page, so that is exactly what we wanted it to do, and it seems to be working perfectly. Let's keep

going next create a new page for test Creation with a form that includes fields for user message, review message, number of requests, and a dynamic section for adding llm providers and models. Yes, please. And as you can see, we are now on the third epic, so we made a lot of progress so far. But again, like I said, this takes time. We're not building a Tic Tac Toe app; we are building a full-stack, real, valuable application, and that takes time. You shouldn't expect to use Pythagora in a number of minutes; you should think about it in tens of minutes and hours. So, it's putting together a plan now. Next, I already know what's coming; it's going to call the Code Monkey to actually write the code, and then we will be testing it. All right, look at all this code being written. Great. Next, it's going to see how do we test it, and then it's going to tell me to test it. All right, lots of steps. Let's do it. I think I already know what most of them are. Start the server. Done. Open

your browser log in log in with the admin or Creator K and then now navigation bar hit the create test link then on the create new test you should see a form with Fields the form is displayed correctly fill it in Great Click the add another llm button select an llm Provider from the dropdown click add another one do it again click remove create test Let's test all of that so back here going to refresh here's the create test tab let's click that all right so we have the user message what is 2 plus2 the review message for number of requests so we'll run it twice we're going to choose gbt 3.5 turbo let's add another llm we're also going to test gbt 4 Turbo preview and just to test removing it we'll add it again and create test so it says after clicking create test it should be logged to the console so let's just verify that so I clicked create test nothing happened let's open up inspect console form submitted right there let's just make

Sure, it looks correct. Okay, so it did submit the form, and so we're good to go. Let's continue, and then on the next step, we're actually going to submit it to the database. Next, implement backend functionality to handle the test creation. Yes, all right, all the new code has been written. Now let's test it. We have some complex tests here, so start the server, open your browser, log in with an existing user, visit test create, fill out the form, click add another llm, expect a result, click the create test, and we should be redirected to the test page. Verify the test look correct, check the database, log out, visit the admin dashboard, and we should see a list of the current users with their roles. We probably don't need to do that again. You should be able to see the newly created test op in the test list. Okay, let's just test everything. So here we are again. I'm going to click the create test tab. Let's do this again. So 4 + 4 review message is eight, number of requests, let's put two.

We're going to test this Open AI GPT-4 Turbo. We'll add it. Let's create a test. Test created successfully. Click okay. Ooh, the test names got deleted, so I ran into a little issue. I see a bug. All of the test names were deleted. I don't know why. So let's just tell it. So all of the test names from the example test were deleted. They show up as blank. Okay, so it's trying to figure out what the bug is, and I can see it's adding some logging, which is good. So now I have to start the app. Let's refresh the page. The test names are not there, but let's just go through one more time creating a new test. So I'll do 2 + 2. Review message 4. Number of requests. Let's do one, and I'll just create it like that. Test created successfully, and here we go. So none of them have names. So now copy back and logs, and going to hit send. So it's asking me for front-end logs. I don't have any, so I'm

just going to continue without logs. Please add any additional feedback. So, I'm going to say the names of the test were gone already when I tried to recreate the bug again. The names of the test should be located at/test. Let's go. Okay, so identified the problem. Here's the solution. So, most likely what I'll have to do is delete all of the tests and try to do it again. Okay, you can reproduce the bug like this. Navigate to the test list page after running the script to populate the database with sample test data. Yep, you will observe that the test names are missing, appearing as blank entries on the list. This issue likely stems from a sample data not including the name field for each test, which is required by the test model. All right, let's give it a try. Start the app, refresh the page. Oh, all the names are there now. Look at that. All right, so I think that actually just fixed it. Yes, the issue is fixed. All right, next we have two unnamed tests. One thing that it should do is

ask me for a name, so let's add that. Okay, so the test creation form should ask me for a test name, which it doesn't right now. Okay, so that should be an easy change, and we'll be able to test it pretty easily as well. So, anytime that you add a change like I just did, it's going to update the initial prompt that we used, which is perfect. So, it always knows what it should be doing. Yes, let's accept those changes. Okay, do you accept these changes in the project specification? Yes. Okay, now it's writing the code to actually do it. So, here's a tip: Pythagora works best if you give detailed feedback. That's always important, not only for human engineers but obviously for AI engineers as well. All right, let's test it again. Visit the creation page, and now we should have a test name field. So, here it is. Refresh the page. Create test. There's the test name. Test test name. User message 1+1. Review message 2. Number of requests 1. Create the test successful, and there it

is perfect. Going to go ahead and just test deleting one more time. That worked. Everything works wonderful. Update the test list page to display the newly created test with their basic information. So, we've already done that, so we can skip that task. Implement the backend Logic for test execution. Yes, that's an important one. Let's go ahead and do that now. We are on, as you can see here, the fourth epic. If I click over to the progress tab, you can see exactly what we're doing and how we are going to progress through this. So, next, Implement real-time progress tracking for test execution. That will be next. Now, back to testing. All right, let's read what we have to do. Start done. Log in admin. Navigate to test create. We're going to create the test. We're going to run the test. Then we're going to navigate to/test, and it should have a list of the tests, including the newly created sample test. Locate the sample test in the list and click execute. Okay, refresh the page after a few moments. The status should be completed. Let's do it. This is awesome.

all right, so we're going to create a new test. Test name: 1+1. User message: 1+1. Review message: The answer is two. Number of requests: one. GPG: 3.5 turbo. Create the test. Test created successfully. There it is. Now we have this execute button. Let's go ahead and click it. Test execution started and immediately completed. Wonderful. Okay, so step nine is a little bit confusing. It says click on the sample test to view its details, but we don't have that yet. So I clicked over to the progress tab just to check if that functionality is going to be implemented in a future task, and there it is. Task four: update the page with the details. So we can just continue. Everything works. Next: Implement real-time progress tracking. Sure. All right, now back to testing. Okay, let's read through how to test it. So start the app, log in, visit the creation page, create a test, click the run test button, and we should see a test progress tracking should be listed with pending. Click on the execute button, observe the

progress bar refresh the page occasionally and then wait till it's complete. Easy, let's do that. So, in fact, I don't really need to create a new one, but we'll do it anyways. I can just execute one of these, but let's keep testing it. So, I'm going to call this 2 + 2 user message 2 + 2 review message answer is four number of requests one. Let's create the test. All right, and execute. And it completes instantly. Okay, so this happened immediately. So, it might be working, it might not, but it's good just to make sure. So, let's just debug it together with Pythagoras. So, I'm going to say there's an issue. So, I'm going to say the progress bar didn't appear. It just went straight to completed. That might mean it just happened so quickly I didn't see it, but I want to just verify that the logic to send the test to the LLM is actually happening. Okay, so now it's saying do another test with multiple LLMs. So, the last couple times I

didn't do that. Let's do it this time. It's writing some logging code just to make sure it's working properly. Okay, so to reproduce this issue, create a new test with multiple LM requests using the test creation form. Okay, so let's do that. Start the app, create test, call this 3 + 3 user message, 3 + 3 review message, answer is 6, number of requests one. We're going to add another LLM, we're going to test GPT-4 Turbo, create the test. All right, then let's execute it. Oh, there it goes, so we actually did see it that time. Okay, good, good, good, good. So let's go back. So maybe it's because we didn't have multiple LLMs running, but let's say copy backend logs, send copy frontend logs. We don't have any, so we're going to continue without logs. An additional feedback, I saw the progress bar this time. Maybe it only shows the progress bar when I use multiple LLMs, but it's still not clear if the code is actually using the LLMs.

to test or not. Can you verify based on the logs and the humans' feedback? We can confirm that the test execution is indeed happening. Great. Yep, there's the test. We can see the progress being updated. Wonderful. And we should add more detailed logs. Great. Breaking down the test now. It's updating the code. Okay, so now reproduce it again. We're going to try it again. So first, start the app, create a new test. We'll call this 4 + 4. User message 4 + 4, and answer is eight. Number of requests one. We'll give it another LLM turbo. Create the test. Great. And execute. Okay, so it's definitely showing the progress bar, but oh, there we go. It slowed down a little bit, so we can actually see it. Great. So now let's switch back, copy back-end logs, send it over, continue without front-end logs, and then the feedback. So let me just check the database one more time. Okay, so looking at it, it's actually storing now. So let's tell it. So it seems to be working now. I see the

test results in the database. Okay, yes, the issue is fixed. All right, and everything works. Let's continue to implement the evaluation process using anthropic. Yes, okay, so now it's doing the anthropic piece. Great, and remember at the beginning we gave it the anthropic API key as well as the OpenAI API key. All right, let's get back to testing. Start the server. Done. Log in. We got all that. Create a new one. Navigate to test. Click execute. Wait for a few minutes to allow the execution to happen. Scroll down to the section displaying the LLM responses and the evaluations. Verify that the evaluations are correctly displayed. But I think that's going to happen on the next step, so I think we can test most of this. So back here, refresh the page. Let's do create new test. Test name 5 + 5. 5 + 5. Review message. Answer is 10. Number of requests one. We're going to test GPT 3.5 turbo. Add another LLM. We will do anthropic, and we will do Claude 3.

Sonet and then create test and execute boom. So now we should see it progress. There it goes. Let's refresh the page now. Like I said, we can't click in, so let's just go back and say everything worked. Now it's going to update the test view page so we can actually see the information. All right, so start the app. I know that's number one. Open your browser. Log in. Navigate. Create the test. Navigate to the test page. Click execute. Wait for the execution to complete. Then we're going to click on the test name. Scroll down to results. We got it. Let's do it. Okay, so we're going to go to create tests. Let's call this 6 + 6 user message. 6 + 6 review message. Answer is 12. Number of requests one. Let's add another LLM. Anthropic Cloud 3 son it. Now one thing I notice is it's asking me to select a provider, but it also just lists all the providers in the next drop-down. It's not a big deal, but we'll probably want to

get that fixed up later so create test all right so we have an error and it seems to be on the test page so let's go back there's an issue uh describe the issue there was an error serving your request I'm getting this error after creating a new test when it tries to navigate me to the SL test page I also got that error when first opening the slash test page even before creating a new test okay so reproduce it don't really think I have to do all that much to reproduce it other than go to the test page so it's going to input some logging all right so reproduce it and then we'll copy the backend log start the app refresh the page there's the error that's all we got to to do copy backend logs boom we don't have any front end logs and I don't really need to give it any additional information so there it is the error is occurring in the list. AGS file l38 it's so cool that I can pinpoint it that easily yes I mean if it

adds all the logging, and then we test it, and then it sees the logging. That's exactly the iterative cycle we would go through as Engineers, but now it's doing it for us. All right, reproduce it. So, start the app, refresh the page, and it works. Awesome. So, I'm going to go back before continuing, and I'm going to say yes, the issue is fixed. Now, back to the testing. I got to start the app again, go to the test page. Let's click execute. There it goes, progress. So, status in progress. Everything's looking good. And now, as you can see, all the names are linked, so we should be able to actually go into it. So, let's refresh the page. It's completed. Click into it, and there it is. Oh my God, this is so cool. All right, all right. So, test name, all the information we put in, results GPT 3.5 turbo response, one evaluation true. Then for Claude 3, Sonet 6 + 6 = 12, evaluation true. This is amazing. Look at this. We already have created so much in this.

benchmarking application. So, yeah, everything works all right. So, everything works. Oh, this is so cool. Let's see. All right, next, create a test view page that displays detailed information about a specific test. We already have that. Let's skip it. Implement a bar chart. Okay, yes, we definitely want that code monkey doing its thing. Now, check out this reworking tab. What that means is it's basically doing self-reflection. It found that it didn't really like the first type of or it didn't really like the first code it wrote, so it's taking another look and it's rewriting it. And done. Now, it's updating the test routes.js file, reviewing it, and should be done any second now. Back to testing. So, we already know a lot of the steps that it's going to tell us to test with. Okay, so let's start it up. We already know we're going to have to do that, create another test, test it, run it, and inspect the chart. Great. Maybe I don't even need to go through all of that if I just refresh this page. Yep, the chart's

already there. Look at how cool this is. Okay, so we're done. We're going to skip past all those other steps because we don't need to test those. Everything works. Let's keep going. Okay, so next we're going to add the ability to actually publish the test page. So let's do that. So this will allow me to share the test results with anybody who wants to see it. And as a reminder, if you ever wanted to go back and start from a previous step or you ever wanted to add more functionality, you can easily do both with Pythagora. So here we go, Code Monkey writing more code, and now you can see we are almost done, and we are on the fifth of six epics. Okay, so now it's going to tell us to test it. All right, let's see what we got to do. We know all these steps. Let's go ahead and click Start right away, click on the test name, and then click Publish. Okay, so we are here. Let's click into one of them, and we got another error. There was an error.

serving your request. Okay, so let's try to fix it. So when I click on a view test details page, I get this error. All right, so of course, it's going to ask me to reproduce it. It's going to add logging, and let's wait for it to finish that. And to me, the debugging is actually the coolest part, just describing what the issue is, reproducing it, adding a bunch of logs, and then it figures it out and fixes it. It's been pretty seamless so far. Okay, so we're going to start up the app. I'm going to refresh this page. Still getting the same error. Perfect. Copy back and logs, and then click Send. We don't need front-end logs, and I don't need any additional feedback. And we can clearly see where the problem is. I mean, that's amazing. Now it's going to attempt to fix it. We're going to test it again, make sure it works, and then we're going to continue. So to reproduce it, let's do it. So start the app, refresh the page, and

It works okay, and there's the published button, but before we test that, let's go back and just say the issue was fixed. Now we will see the publish button. Let's go ahead and publish it. An error occurred while publishing the test. Okay, so let's copy that. I'm going to refresh the page. Oops, oh right, I have to start the app again. Let's refresh the page. Let's click publish test. An error occurred while publishing the test. So, I'm going to copy that. I'm going to refresh the page, and it does say published, interestingly enough. So, I think there's a bug, but let let's tell it exactly what happened. All right, so it seems to have worked, but the popup after clicking publish said an error occurred. Okay, so here it is. Start the app, refresh the page. This one's already published, so let's go back to the test page. I'm going to click another one, go down to the bottom, click publish test. An error occurred. Okay, so it's still there, and I'll refresh the page, and it says status.

publish, so it did work. Same thing, copy back-end logs and continue. No front-end logs, no feedback needed. All right, so it seems like it found the issue. Oh no, it's adding more logs. Okay, so it needs a little bit more information. So this is the first time we've actually needed to do this twice. So this is going to be interesting. Okay, so let's recreate it again. Start the app, refresh the page. We're going to need a new one. So let's go back to test, go to 44, publish, an error occurred, click okay, go back, copy back-end logs, send it. Don't need any front-end logs, don't need an additional feedback. Okay, we can see that the back-end is functioning properly. So we did know that. That's good. And now it's adding more logs to identify why it's still getting an error. So everything's working, it's just telling us the wrong thing. I need human intervention. Open the browser's developer tools and right-click and hit inspect. Okay, console, and there it is. So we see some issues.

here all right so I'm going to copy all of those front-end logs. Provide any logs or errors. Okay, continue. So, but it says copy backend log. So, let's redo it again. Test. Let's go down to two of two publish. There it is. All right. Now I'm going to inspect console. There it is. Okay. So, there's that error. Let's see if there's any more info in there. Yeah. So, we'll get all of that info. Copy backend logs. Continue. This time we do have front-end logs. Then let's hit send and then continue without feedback. Okay. Now we can clearly see. All right. Perfect. Now it is fixing the bug. All right. Reproduce the issue. Great. Let's try it out again. We're going to start the app. Refresh the page. Here's one of one. Let's publish the test. And it worked. Perfect. All right. So, yes, this fixed the issue. Yes, it says publish. Now everything works. Beautiful. Next, implement the backend logic of publishing. Yes, go ahead. So, we are going to click the publish button. The publish test button should be removed. Log out.

and then visit, and then we should see that published test. All right, so even though it says to check that, if we just look at the progress bar, that's actually going to be the final epic, is actually making the homepage display all that information. So for now, we just click everything works. Now we are on the final step, update the homepage to display the list of published tests. And again, this means we're not logged in, but we're able to see the test, thus they are published. All right, so that should be done determining how to test it. It's going to give me all the steps, but we don't need to follow all the steps. Let's just start the app and just see if we have a bunch of tests. So I'm going to log out, go to the homepage. There they are. Boom, look at that. Let's click into one of them. Ah, so it won't show us the actual test results. All right, so even though it does give me an issue, if I look at the progress, I can see that's

the very next thing that we're going to be doing, so I'm just going to say everything's working and let's continue. And then next, what we should be able to do is reload the test View page that has been published as a non-authenticated user, and we should be able to see it. So yeah, sometimes what will happen is it tells you to test for something that it hasn't written yet, and that's okay. Just keep an eye on the progress bar so you can see what's coming, and if you see that that functionality is coming, go ahead and just continue. Everything's working. All right, let's give it a try. So let's start the app, come back here, let's refresh the page, and there are the details. I'm not logged in. We can see that because it says log in right there, and we have all of the results. Perfect. Okay, let's continue. Everything works. Now it's creating the read me. Awesome. Tells you exactly how to set it all up, how to reproduce all of these things that we just did. All right, so it

says next add navigation links, but we already have that. I tested it, so we don't need it anymore. So, let's skip. And then the last step, are you satisfied? So, it's asking us, yep, app works great. Let's submit now. Let's deploy it. Here's the tab for deploying. Want to deploy your project? One click. Go ahead and deploy. Look how easy this is. All right. And as it's deploying, and it just finished, I will give you this link. I'll drop it in the description below, and you can try it. You can try this application if you want. So, click open URL. Let's open it. And there it is, deployed live on the web. Very easy, awesome. And all of the data is still there, which is really cool. So, that's it. We're done. I just created a full-stack application, a very valuable one, one that runs benchmark tests against different LLMs. It has authentication, it has admin, it has publishing tests to the web with non-authenticated users, hundreds, maybe even thousands of lines of code, and I wrote absolutely zero of them. And in

fact. If I click this little menu right here, I can see exactly how many lines of code were written: 1,637 different files, and again, I wrote none of them. So, very cool, amazing work by Pythagora. Thank you again to Pythagora for working with me on this video. This has been awesome. I know this is on the longer side of the video, but I wanted to walk you through what it looks like to actually build a full-stack application. Now, if I did want to continue and build that more functionality, I could easily do that as well. Thanks again to Pythagora. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

This AI Coder Is On Another Level (Pythagora Tutorial)

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

The video demonstrates building a full-stack LLM benchmarking application using Pythagora, with 1,600 lines of code in under 2 hours.

The application aims to test different large language models against a set of test questions, featuring user authentication, an admin dashboard, and the ability to create and publish tests.

Pythagora's iterative approach involves different agents handling various development roles, making the process efficient and collaborative.

Updating User Model and Creating Admin User

The user model is being updated to include a role field with a default value of viewer.

The developer agent is writing and displaying the new code, with a progress bar and logs for tracking.

The application is being tested by registering a user, verifying the database entry, and creating an admin user through a script.

Building a Real-World Application with Code Monkey

The Code Monkey agent is updating the code in real-time while explaining each step of the process.

The application being built includes an admin dashboard with user role management, test data population, and a delete button for test entries.

The workflow is smooth and efficient, allowing for quick task progression and testing, ultimately creating a comprehensive real-world application.

Testing and Debugging Test Creation Functionality

The script for populating tests was initially recreating five tests instead of adding them, causing an issue.

The code was updated to add five new tests without deleting the existing ones, fixing the problem.

The test creation form was successfully implemented, but a bug caused the test names to be deleted, leading to further debugging and logging to identify the issue.

Debugging and Testing Test Execution Flow

The test names were initially missing due to the sample data not including the name field, but refreshing the page after the app start fixed the issue.

The test creation form was updated to ask for a test name, and the backend logic for test execution was implemented.

Real-time progress tracking for test execution was tested, and the progress bar appeared when using multiple LLMs, but it was unclear if the code was effectively utilizing the LLMs.

The test execution is happening successfully, with progress updates and detailed logs being added.

The issue with storing test results in the database has been fixed, and the evaluation process using Anthropics is being implemented.

However, an error occurred on the test page, which was easily pinpointed and resolved, leading to successful test execution and accurate result display.

Building and Debugging a Test Page Application

The application is progressing well, with features like a test view page and a bar chart being implemented.

The code monkey is self-reflecting and rewriting code for better functionality.

Debugging has been seamless, with the application identifying and fixing errors in real-time, but a persistent issue with the publish button has required additional logging and human intervention for resolution.

Building a Full-Stack Application with Pythagora

The process involved copying backend and frontend logs to troubleshoot and fix a bug, leading to a successful test publication.

The application then implemented backend logic for publishing and updated the homepage to display published tests, allowing non-authenticated users to view them.

Finally, the project was effortlessly deployed live on the web, with a total of 1,637 lines of code written by Pythagora, showcasing the ease of building a full-stack application.