Pythagora로 풀스택 LLM 벤치마킹 애플리케이션 구축하기

### Pythagora로 풀스택 LLM 벤치마킹 애플리케이션 구축하기
* 🚀 이 튜토리얼에서는 **1,600줄의 코드로 Pythagora를 사용하여 2시간 이내에 풀스택 LLM 벤치마킹 애플리케이션**을 구축하는 방법을 보여드리겠습니다.
* 💻 프로젝트는 **VS Code에 Pythagora 확장 프로그램을 설치**하고 **Node.js를 비롯한 필요한 코드 프레임워크**를 설정하는 것으로 시작됩니다.
* 📝 저는 사용자가 미리 정의된 일련의 질문에 대해 다양한 대규모 언어 모델(LLM)을 테스트할 수 있는 **벤치마킹 애플리케이션**을 만들어 단순한 프로젝트가 아닌 합법적이고 유용한 애플리케이션임을 강조할 것입니다.
* 🔑 사용자 인증 및 관리자 대시보드를 포함한 앱의 기능을 자세히 설명한 후 개발 프로세스를 안내하는 광범위한 프롬프트를 **Pythagora에 붙여넣겠습니다.**
* 🛠 Pythagora의 에이전트는 개발팀에서 다양한 역할을 맡게 되는데, **아키텍트 에이전트는 Node.js 및 MongoDB와 같은 기술을 결정**하고 **스펙 작성자는 프롬프트의 복잡성을 확인**하는 역할을 맡게 됩니다.
* 🔍 Pythagora와 상호작용할 때 **API 키 등 필요한 입력**을 요청하고, 작성 중인 코드에 대한 **실시간 업데이트**를 제공하면서 디버깅을 도와줍니다.
* 📊 이 과정에서 **테스트를 실행**하고 사용자 등록이 제대로 작동하는지, 관리자 대시보드가 올바르게 표시되는지 확인하여 앱의 기능을 검증합니다.
* 🔄 반복적인 개발 주기를 통해 **역할 관리 및 페이지 매김과 같은 기능**을 추가하는 등 지속적으로 개선하여 궁극적으로 실제 요구 사항을 충족하는 강력한 애플리케이션을 완성할 수 있습니다.

### Pythagora로 풀스택 벤치마크 테스트 애플리케이션 구축하기
* 🛠 이 프로세스는 **테스트 이름이 삭제되는 초기 문제가 발생**하여 버그 조사를 촉발한 **Open AI GPT-4 Turbo를 테스트**하는 것으로 시작되었습니다.
* 🔍 팀에서 문제를 해결하던 중 **샘플 데이터에 필수 이름 필드가 포함되어 있지 않아 테스트 이름이 누락**되었다는 사실을 발견했습니다.
* ✅ 문제를 해결한 후 테스트 생성 양식을 업데이트하여 **테스트 이름을 입력**하도록 하여 사용자 환경을 개선했습니다.
* 📊 이 팀은 **테스트 실행을 위한 실시간 진행 상황 추적**을 성공적으로 구현하여 사용자가 테스트 상태를 실시간으로 확인할 수 있도록 했습니다.
* ⚙ 이후 테스트 결과, 테스트가 성공적으로 게시되었음에도 불구하고 **오류가 발생하는 게시 기능의 버그**가 발견되어 추가 디버깅을 진행했습니다.
* 📈 몇 번의 반복과 개선 끝에 애플리케이션이 완벽하게 작동하여 **사용자가 원활하게 테스트를 생성, 실행 및 게시**할 수 있게 되었습니다.
* 🌐 마지막 단계는 웹에 애플리케이션을 실시간으로 배포하는 것으로, **코드를 직접 작성하지 않고도 다양한 LLM에 대한 벤치마킹 테스트에서 애플리케이션의 기능**을 보여 주는 것이었습니다.
* 🎉 이 프로젝트는 다양한 파일에 걸쳐 **총 1,637줄의 코드**를 작성하여 복잡한 애플리케이션을 효율적으로 구축하는 Pythagora의 강력한 기능을 보여주었습니다.

Pythagora를 사용하면 2시간 이내에 풀스택 LLM 벤치마킹 애플리케이션을 손쉽게 구축할 수 있습니다.

watch me build a full-stack LLM benchmarking application, 1,600 lines of code in under 2 hours using an AI partner. Today, we're going to be using Pythagora to do all of this, and I've made multiple videos about Pythagora in the past, and you've always had great feedback about those videos, so we're doing it again. So, I'm very happy to be partnering with Pythagora once again on this video. So, let's just get into it. All right, so let's start building. So, we have the Pythagora extension installed into VS Code. I have the code framework I'm going to be using, which includes Node already installed. If you don't have those installed, go ahead and install them. And here is what Pythagora looks like. So, we're just going to dive right in. Let's click Create New App. The first thing it's going to ask for is a name. So, what is the project name? And today, what we're going to be building is something that I think I'm going to be

using all the time, which is a benchmarking application. And remember, when we're building with Pythagora, we're not building toy applications. This isn't like a to-do list or anything like that. This is a legitimate full-stack valuable application that doesn't take minutes to build. It might take hours, but it is something that we will actually use in our day-to-day lives. And so, what we're building is a benchmarking application, so something where I can give it a set of LLM test questions, and it will automatically test different large language models against those test questions, essentially a benchmark. And I want to use the benchmarks that I use in all my videos, so I'm going to be doing that. But this is going to be more generic than that. I can just decide to use those benchmark questions in this application. And I've already written up a pretty extensive prompt to start this project. I'll drop the prompt in the description below, and I'll just go over it briefly here, but it's pretty long, so not going to go through it completely. So

first the project name we'll just call it Benchmark. And here it says describe your app in as much detail as possible. And you're going to see how much detail we really describe it in. So I'm just going to go ahead and paste it in here. And all of this is describing how this benchmarking application will behave. So I'm just going to go through it very quickly. I want to create an app that will enable users to create tests for llms, run them, and publish the results. First, it needs a homepage with a welcome message, a list of published tests. Users should be able to click on a test that opens a test view page. It needs authentication, including administrators. It needs an admin dashboard, a page with a list of created tests, and a creation page for a new test. Here's the description about how I actually want the test created, more about what the llms will actually do with those tests, all the way down here, which large language models to use, and then finally a way to share it. So again, I'll drop this prompt in the

description below. You can also just click Start an example project if you don't know what you want to build yet and just want to try it out, or you can import an existing project. But since I already wrote this out, we're just going to click the little send button right here and get started. And what Pythagora, as I mentioned, is really good at is building full-stack real applications, valuable applications. And so the way that they do that is they have different agents that take on different roles as kind of a developer or development team that you manage. So we have the spec writer and the architect agent. So the spec writer is checking the complexity of the prompt. The architect agent is starting to plan it, asked me if I want to use the Node Express template. And since that is the framework I'm going to be using, I'm going to click yes. So the architect agent decides which technologies to use. So let's see, Node.js, MongoDB, Express, Express session, and so on, all the way down. So they're really choosing the full-stack application to

use architect agent made sure we have MongoDB installed. Here's some output. So imagine we were doing this through the terminal. This is exactly what we would be seeing there. Now Pythagora is asking us for input required on ENV. So ENV line 9. Please open the file. It does that automatically and it also highlights the line that we need to adjust. So what we need to do is insert your API key here. And that's what it's asking us to do. So I'm going to go grab that right now. All right. So I've grabbed first the OpenAI API key and I will be revoking this API key before publishing the video. Next I know it's going to ask for the Anthropic key. So I'm going to grab that as well. So both keys I'm going to revoke before publishing. There it is the Anthropic key. I forgot to remove this space right there. So let me go ahead and do that. I'll save. And now that I've gotten everything that it's asking I'm going to

go ahead and press continue. Okay, yep, it's asking for line 10. I've already done that, so I'll hit continue again. So, this is the interaction with Pythagora. It is basically doing all the work for me, and then when it needs me to intervene in any way, whether that's helping debug something or inputting custom API keys or anything else, it will ask me to help. It will tell me where to look to help and what it needs from me. And this is, if you remember from the last two videos about Pythagora, this is the interaction cycle with it. It's just very iterative, and it does a bunch of work, then it says, Hey, test this for me, and then if there are any issues, you describe the issues, and it walks you through debugging them. So, let's look at what's done so far. So, we have the tech lead agent creating the development plan, creating tasks, and it's putting together an entire development plan. It's not just going and building. So, developer agents starting new task with description.

update user model to include a role field with default value viewer. Do you want to execute the above task? So, you have three options, and I'll go over this the first time, but on subsequent times, I'm just going to go ahead and make the decision. So, do you want to execute the above task? You can either say yes, you can edit the task, and so you can give it more instructions or change how it is going to be done, or you can skip the task in case you don't want to do it at this time. So, right now, we're going to go ahead and press yes. So, the developer agent is thinking, writing code, and as it's writing new code, we're going to be able to see all the new code written right here. We're going to be able to see the diffs. It's really neat. And then also, you can see a progress bar right up here, so you can see the progress as you progress through building the applications. There's a logs tab. There are no logs at the moment, but any logs, front end or back end, will be

here you can also see the more detailed progress information in the progress tab. So you can see all the different tasks that were created, all the epics, and you're basically working through it step by step. Then the deployment tab, so we're not quite at that step yet, but we'll get there. Okay, now back to here. This is the tag that tells us we need to get involved, human test. So figuring out how to run the app, determining how to test the app, and now it's giving us instructions to go ahead and make sure everything's working so far. So action, start the server using npm start. Usually, you can do that through terminal, but it also has this nice little button with the rocket ship saying start app. You just click that, and it starts the app for us. All right, so we have the npm up and running. Just to make sure it is, we can click over the logs tab. There it is, server running at Local Host. So let's go ahead and copy that. I'm going to switch back to the development.

tab then I'm going to open up a browser. I'm going to type in Local Host, hit enter, and there it is, it's up and running. So now we've verified that that works. Let's switch back to VSS code, and we have three options at the bottom: everything works, make a change, or there is an issue. So far, there are no issues, so we can keep going. Next, it's saying we need to register, so let's test registering. So it actually wrote a lot of code already, and actually, you can see a lot of the code right here. Here's like off routes, for example, user.js. And again, as I mentioned, as you build out the application, you're going to see more and more code over here. And if we click the files tab, we will actually see a lot of these files being written as well. Next step two, follow the steps. We're going to grab this URL, very common auth register. Okay, so I just put an AM Burman and just a quick password. We're going to click register login, and it seems to

have worked there. There's the log out button, so we are logged in, so we verified that. Okay, so next step three, fill out the username and password. We've already done that, so it did work. Step four, open your MongoDB client or MongoDB Atlas and navigate to the database used by your application. Okay, so I've opened up MongoDB Compass to just check the database, make sure everything looks correct. Let's go back to make sure we know what we're looking for. So we're going to look for a user's collection and a newly created user document. It should have a username, a password that is hashed, and a role with a value viewer. So let's take a look. All right, so within Pythagora, we have users, and there it is. There's the user with the correct role, so everything is created properly. Let's keep going, so switch back. Everything works perfect. Yeah, so we've had no issues so far. This is great. Starting a new task with description create an admin user creation script that inserts the first admin user into the database. Do you want to execute? Yes. Okay, there so

it updated the code. Not only can we see the updates to the code, but we can actually see the if we were to click any of these buttons right here, but I'm not going to do that right now. So these are the different files that were just updated, and now it's asking for something from us. So input required on this line and create admin user right there highlighted already open. So input required provide the admin username and the password. Okay, so I came into EMV. I updated these two fields, and we should be good to go. So let's hit continue. It's asking me to do that for the password. I've already done on that, so let's hit continue again. And now it's asking for the database URL, but I already have that, so it's all good. Let's keep going. Okay, can I run the command npm run create admin with a 60-second timeout? Yes, you can. There it goes. Admin user created. Perfect. Determining how to test the app. Great. All right, now we got a bunch of tests that we need to run ourselves, so

action update the EnV with these admin credentials. We've already done that. Run the script to create the admin user using npm run create admin. In so that actually already ran, so I don't need to do that again. You could run it again, but you don't need to start the server. We already have one started. I go ahead and click it again. Open your browser. So next, we're going to log in. Let's start the app, and we're going to type in our login and password. And there we go. Now we're logged in, and we will go on to the next step. Okay, log in. It did work. Verified your login. Yep. Uh, open your browser to the admin dashboard. Let's do that. Page not found. Yep. Expected result. You should see an error page not found. Perfect. Uh, everything works correctly. Let's keep going. Starting new task. Implement an admin dashboard page. Yes, go ahead. Okay, once again, writing a bunch of code, and all of this code is going to show up on the left side over here, and not only is

it is writing the code, but it is explaining what it is doing, why it's writing the code as it goes, breaking down the task into steps. Code Monkey agent, now we're going to start seeing the code get updated over here. Yep, there it is, and everything's switching automatically to the code that is updating in real time. Okay, once again, we see all the diffs here, so lines created, lines removed, and of course, let me just show you when it's done. We can just click into one of them, and we can see the diffs just like that. Okay, testing again and again. This is just the iterative cycle that we go through. So, expected result, start the server using npm start. We could do that by clicking this button right there. Open the browser, log in with the admin credentials. Let's do that again, and there we go, and there's that new admin dashboard link at the top. Let's click it, and there it is, a very simple admin dashboard. It tells us the usernames and the role. Step three, login. Yep, step four, navigation bar admin.

dashboard yep and okay so everything works. Let's click it and keep going. Starting new task. So add functionality to the admin dashboard to allow the admin to change user roles between viewer and Creator. Yes, all right, right in code, and look up here. You can see we are progressing through the different steps of the application we're building, and this is one of 1, 2, 3, 4, 5, 6 different epics. And then within each epic, you can see all of the progress we're making. So here it is, Code Monkey agent working. All right, it's done, human test. We knew it was coming, determining how to test the app. All right, we got a lot of tests to do this time, so let's see. Start the server, done. Open the browser, log in. Let's just read through, and we can do it all at once in the browser. Visit admin, click the admin dashboard, refresh the admin dashboard. Oh, nope, sorry, one step back. We will change it from viewer to Creator, then refresh the dashboard, make sure things are changed, and let's stop there. So here

we are, and we can see the new functionality. There's the role. We're going to change the role of M Burman from viewer to Creator. We're going to click save. Roll updated successfully. Let's refresh the page, and we can see it has been updated successfully. Awesome. And now it says change it back to viewer. So let's go ahead and do that. Click save. Successfully saved. Reload. And there it is again. Okay, so now it says let's log out and then log into the non-admin user. Then it's saying let's test going to the admin page to make sure we do not have permissions. There it is. Access denied. Perfect. And then it's saying log out again and just make sure we can log in with the admin. So I'm going to log out, log back in with admin, admin dashboard. And there it is. So everything works perfectly. Let's keep going. Next, implement access control to ensure only admin users can access the admin dashboard. So we already did this one. Sometimes it'll suggest things we've already done, so that's okay. That's why

There is the skip task button right there, so we're just going to skip it. Next, create a new page for listing tests with a table displaying test names and creation dates. Yep, we definitely want that. Let's click yes. All right, so it wrote a bunch of new code. Now we test it again, so start the server. Easy click open to off login. Then we're going to log in with the valid credentials, and we are going to see the test link in the navigation bar. So let's do that. So I'm logging in as a regular user. There's the test tab. Click it, and no test available. So that worked. All right, everything works. Let's keep going. Implement a script to populate the database with sample test data. Yes, please. Okay, there it goes. All right, Code Monkey get to work. Populate test data.js. There it is, pops right open, and package.json reviewing. So we could see again all the diff right here. Can I run the command npm installed. NV with 60c timeout. Yes, you can. Can I run the command populate test. Yes, all.

right now back to testing. Start the server, click run the script to populate the database. We already did that. Open your browser, log in. We're already logged in basically. Look at the test page. There it is. Perfect. All right, that is definitely working. Everything works. Add a delete button. Yes, and so you get into this nice flow with your developer agent and your Pythagore team in general, and it just is super easy. So write in the code once again and let's test it again. All right, let's read through how it wants us to test it. Start the server, log in. Yep, log in using an admin account. Okay, visit the test page, click delete, confirm, refresh, make sure it's deleted, and let's do it. Okay, so I'm going to log out. I'm going to log in as the admin, go to test, and there are the delete buttons. So we're going to delete the code completion test. Let's click delete. Okay, and it's gone. I'm going to refresh the page just to make sure. Great, let's delete one more text summarization.

test okay gone refresh done all right easy peasy. Let's keep going all right let's do something else. So Implement pagination yes. I mean this is so cool cuz it's just so easy and now that I am in the flow of using it I can kind of almost do it faster. I can skip some steps. I know I don't necessarily need to log in every single time uh but I just want to keep an eye on what it's telling me to do and what it's telling me to test and by the way look how much code has already been written. All of these different files and so yeah this is so much more than just a simple Todo app. This is a real-world application that we are building from the ground up using pythagora. All right let's read through how to test it again. Open your browser visit login login with the admin credentials visit test. We're already there if there are more than 10 test scroll to the bottom of the page. The expected result pagination control should be there click on the page number.

verify that each page number displays the correct number of tests. Click through all the pages. Click on the delete button. Confirm it deletes all right. So basically just confirm everything again. Let's do it so back here going to refresh and it only has three tests. So I'm going to go ahead and run that script a couple more times just to get more than 10 tests in here. Back to Pythagora. Open up terminal. So we're going to run npm run populate test. Hit that once. Good. Let's do it again. And let's just do it one more time for good measure. So now we should have 15 additional tests in there. So let's switch back to the UI. Hit refresh. Okay. So it recreated all five but it didn't add five. So that's a problem. And let's just fix it. So I'm going to come back here. I'm going to say there is an issue. So I'm simply going to describe what the problem is. I'm trying to create more example tests but each time I run populate tests which is the npm script.

it recreates five rather than adding five. Can you make the script add five in addition to whatever examples already in there? Yeah, and the solution will be not deleting the existing test. Perfect, so it's rewriting that part of the code. All right, and let's run the script one more time. I'm going to start the app again, let's refresh the page, and there we go, we have a bunch of new tests. So that was so easy. I'm going to switch back and add 10 more, so that means we will run it twice, once, twice, and now because we have so many tests, we should see the pagination. All right, there it is, perfect. So yes, the issue is fixed. Back to the functionality we were adding, so I had to start the app again, refresh the page, page two, and that does seem to work perfect. Okay, now let's just try deleting one. Okay, page one, page two, and it always has 10 on the first page, so that is exactly what we wanted it to do, and it seems to be working perfectly. Let's keep

going next create a new page for test Creation with a form that includes fields for user message, review message, number of requests, and a dynamic section for adding llm providers and models. Yes, please. And as you can see, we are now on the third epic, so we made a lot of progress so far. But again, like I said, this takes time. We're not building a Tic Tac Toe app; we are building a full-stack, real, valuable application, and that takes time. You shouldn't expect to use Pythagora in a number of minutes; you should think about it in tens of minutes and hours. So, it's putting together a plan now. Next, I already know what's coming; it's going to call the Code Monkey to actually write the code, and then we will be testing it. All right, look at all this code being written. Great. Next, it's going to see how do we test it, and then it's going to tell me to test it. All right, lots of steps. Let's do it. I think I already know what most of them are. Start the server. Done. Open

your browser log in log in with the admin or Creator K and then now navigation bar hit the create test link then on the create new test you should see a form with Fields the form is displayed correctly fill it in Great Click the add another llm button select an llm Provider from the dropdown click add another one do it again click remove create test Let's test all of that so back here going to refresh here's the create test tab let's click that all right so we have the user message what is 2 plus2 the review message for number of requests so we'll run it twice we're going to choose gbt 3.5 turbo let's add another llm we're also going to test gbt 4 Turbo preview and just to test removing it we'll add it again and create test so it says after clicking create test it should be logged to the console so let's just verify that so I clicked create test nothing happened let's open up inspect console form submitted right there let's just make

Sure, it looks correct. Okay, so it did submit the form, and so we're good to go. Let's continue, and then on the next step, we're actually going to submit it to the database. Next, implement backend functionality to handle the test creation. Yes, all right, all the new code has been written. Now let's test it. We have some complex tests here, so start the server, open your browser, log in with an existing user, visit test create, fill out the form, click add another llm, expect a result, click the create test, and we should be redirected to the test page. Verify the test look correct, check the database, log out, visit the admin dashboard, and we should see a list of the current users with their roles. We probably don't need to do that again. You should be able to see the newly created test op in the test list. Okay, let's just test everything. So here we are again. I'm going to click the create test tab. Let's do this again. So 4 + 4 review message is eight, number of requests, let's put two.

We're going to test this Open AI GPT-4 Turbo. We'll add it. Let's create a test. Test created successfully. Click okay. Ooh, the test names got deleted, so I ran into a little issue. I see a bug. All of the test names were deleted. I don't know why. So let's just tell it. So all of the test names from the example test were deleted. They show up as blank. Okay, so it's trying to figure out what the bug is, and I can see it's adding some logging, which is good. So now I have to start the app. Let's refresh the page. The test names are not there, but let's just go through one more time creating a new test. So I'll do 2 + 2. Review message 4. Number of requests. Let's do one, and I'll just create it like that. Test created successfully, and here we go. So none of them have names. So now copy back and logs, and going to hit send. So it's asking me for front-end logs. I don't have any, so I'm

just going to continue without logs. Please add any additional feedback. So, I'm going to say the names of the test were gone already when I tried to recreate the bug again. The names of the test should be located at/test. Let's go. Okay, so identified the problem. Here's the solution. So, most likely what I'll have to do is delete all of the tests and try to do it again. Okay, you can reproduce the bug like this. Navigate to the test list page after running the script to populate the database with sample test data. Yep, you will observe that the test names are missing, appearing as blank entries on the list. This issue likely stems from a sample data not including the name field for each test, which is required by the test model. All right, let's give it a try. Start the app, refresh the page. Oh, all the names are there now. Look at that. All right, so I think that actually just fixed it. Yes, the issue is fixed. All right, next we have two unnamed tests. One thing that it should do is

ask me for a name, so let's add that. Okay, so the test creation form should ask me for a test name, which it doesn't right now. Okay, so that should be an easy change, and we'll be able to test it pretty easily as well. So, anytime that you add a change like I just did, it's going to update the initial prompt that we used, which is perfect. So, it always knows what it should be doing. Yes, let's accept those changes. Okay, do you accept these changes in the project specification? Yes. Okay, now it's writing the code to actually do it. So, here's a tip: Pythagora works best if you give detailed feedback. That's always important, not only for human engineers but obviously for AI engineers as well. All right, let's test it again. Visit the creation page, and now we should have a test name field. So, here it is. Refresh the page. Create test. There's the test name. Test test name. User message 1+1. Review message 2. Number of requests 1. Create the test successful, and there it

is perfect. Going to go ahead and just test deleting one more time. That worked. Everything works wonderful. Update the test list page to display the newly created test with their basic information. So, we've already done that, so we can skip that task. Implement the backend Logic for test execution. Yes, that's an important one. Let's go ahead and do that now. We are on, as you can see here, the fourth epic. If I click over to the progress tab, you can see exactly what we're doing and how we are going to progress through this. So, next, Implement real-time progress tracking for test execution. That will be next. Now, back to testing. All right, let's read what we have to do. Start done. Log in admin. Navigate to test create. We're going to create the test. We're going to run the test. Then we're going to navigate to/test, and it should have a list of the tests, including the newly created sample test. Locate the sample test in the list and click execute. Okay, refresh the page after a few moments. The status should be completed. Let's do it. This is awesome.

all right, so we're going to create a new test. Test name: 1+1. User message: 1+1. Review message: The answer is two. Number of requests: one. GPG: 3.5 turbo. Create the test. Test created successfully. There it is. Now we have this execute button. Let's go ahead and click it. Test execution started and immediately completed. Wonderful. Okay, so step nine is a little bit confusing. It says click on the sample test to view its details, but we don't have that yet. So I clicked over to the progress tab just to check if that functionality is going to be implemented in a future task, and there it is. Task four: update the page with the details. So we can just continue. Everything works. Next: Implement real-time progress tracking. Sure. All right, now back to testing. Okay, let's read through how to test it. So start the app, log in, visit the creation page, create a test, click the run test button, and we should see a test progress tracking should be listed with pending. Click on the execute button, observe the

progress bar refresh the page occasionally and then wait till it's complete. Easy, let's do that. So, in fact, I don't really need to create a new one, but we'll do it anyways. I can just execute one of these, but let's keep testing it. So, I'm going to call this 2 + 2 user message 2 + 2 review message answer is four number of requests one. Let's create the test. All right, and execute. And it completes instantly. Okay, so this happened immediately. So, it might be working, it might not, but it's good just to make sure. So, let's just debug it together with Pythagoras. So, I'm going to say there's an issue. So, I'm going to say the progress bar didn't appear. It just went straight to completed. That might mean it just happened so quickly I didn't see it, but I want to just verify that the logic to send the test to the LLM is actually happening. Okay, so now it's saying do another test with multiple LLMs. So, the last couple times I

didn't do that. Let's do it this time. It's writing some logging code just to make sure it's working properly. Okay, so to reproduce this issue, create a new test with multiple LM requests using the test creation form. Okay, so let's do that. Start the app, create test, call this 3 + 3 user message, 3 + 3 review message, answer is 6, number of requests one. We're going to add another LLM, we're going to test GPT-4 Turbo, create the test. All right, then let's execute it. Oh, there it goes, so we actually did see it that time. Okay, good, good, good, good. So let's go back. So maybe it's because we didn't have multiple LLMs running, but let's say copy backend logs, send copy frontend logs. We don't have any, so we're going to continue without logs. An additional feedback, I saw the progress bar this time. Maybe it only shows the progress bar when I use multiple LLMs, but it's still not clear if the code is actually using the LLMs.

to test or not. Can you verify based on the logs and the humans' feedback? We can confirm that the test execution is indeed happening. Great. Yep, there's the test. We can see the progress being updated. Wonderful. And we should add more detailed logs. Great. Breaking down the test now. It's updating the code. Okay, so now reproduce it again. We're going to try it again. So first, start the app, create a new test. We'll call this 4 + 4. User message 4 + 4, and answer is eight. Number of requests one. We'll give it another LLM turbo. Create the test. Great. And execute. Okay, so it's definitely showing the progress bar, but oh, there we go. It slowed down a little bit, so we can actually see it. Great. So now let's switch back, copy back-end logs, send it over, continue without front-end logs, and then the feedback. So let me just check the database one more time. Okay, so looking at it, it's actually storing now. So let's tell it. So it seems to be working now. I see the

test results in the database. Okay, yes, the issue is fixed. All right, and everything works. Let's continue to implement the evaluation process using anthropic. Yes, okay, so now it's doing the anthropic piece. Great, and remember at the beginning we gave it the anthropic API key as well as the OpenAI API key. All right, let's get back to testing. Start the server. Done. Log in. We got all that. Create a new one. Navigate to test. Click execute. Wait for a few minutes to allow the execution to happen. Scroll down to the section displaying the LLM responses and the evaluations. Verify that the evaluations are correctly displayed. But I think that's going to happen on the next step, so I think we can test most of this. So back here, refresh the page. Let's do create new test. Test name 5 + 5. 5 + 5. Review message. Answer is 10. Number of requests one. We're going to test GPT 3.5 turbo. Add another LLM. We will do anthropic, and we will do Claude 3.

Sonet and then create test and execute boom. So now we should see it progress. There it goes. Let's refresh the page now. Like I said, we can't click in, so let's just go back and say everything worked. Now it's going to update the test view page so we can actually see the information. All right, so start the app. I know that's number one. Open your browser. Log in. Navigate. Create the test. Navigate to the test page. Click execute. Wait for the execution to complete. Then we're going to click on the test name. Scroll down to results. We got it. Let's do it. Okay, so we're going to go to create tests. Let's call this 6 + 6 user message. 6 + 6 review message. Answer is 12. Number of requests one. Let's add another LLM. Anthropic Cloud 3 son it. Now one thing I notice is it's asking me to select a provider, but it also just lists all the providers in the next drop-down. It's not a big deal, but we'll probably want to

get that fixed up later so create test all right so we have an error and it seems to be on the test page so let's go back there's an issue uh describe the issue there was an error serving your request I'm getting this error after creating a new test when it tries to navigate me to the SL test page I also got that error when first opening the slash test page even before creating a new test okay so reproduce it don't really think I have to do all that much to reproduce it other than go to the test page so it's going to input some logging all right so reproduce it and then we'll copy the backend log start the app refresh the page there's the error that's all we got to to do copy backend logs boom we don't have any front end logs and I don't really need to give it any additional information so there it is the error is occurring in the list. AGS file l38 it's so cool that I can pinpoint it that easily yes I mean if it

adds all the logging, and then we test it, and then it sees the logging. That's exactly the iterative cycle we would go through as Engineers, but now it's doing it for us. All right, reproduce it. So, start the app, refresh the page, and it works. Awesome. So, I'm going to go back before continuing, and I'm going to say yes, the issue is fixed. Now, back to the testing. I got to start the app again, go to the test page. Let's click execute. There it goes, progress. So, status in progress. Everything's looking good. And now, as you can see, all the names are linked, so we should be able to actually go into it. So, let's refresh the page. It's completed. Click into it, and there it is. Oh my God, this is so cool. All right, all right. So, test name, all the information we put in, results GPT 3.5 turbo response, one evaluation true. Then for Claude 3, Sonet 6 + 6 = 12, evaluation true. This is amazing. Look at this. We already have created so much in this.

benchmarking application. So, yeah, everything works all right. So, everything works. Oh, this is so cool. Let's see. All right, next, create a test view page that displays detailed information about a specific test. We already have that. Let's skip it. Implement a bar chart. Okay, yes, we definitely want that code monkey doing its thing. Now, check out this reworking tab. What that means is it's basically doing self-reflection. It found that it didn't really like the first type of or it didn't really like the first code it wrote, so it's taking another look and it's rewriting it. And done. Now, it's updating the test routes.js file, reviewing it, and should be done any second now. Back to testing. So, we already know a lot of the steps that it's going to tell us to test with. Okay, so let's start it up. We already know we're going to have to do that, create another test, test it, run it, and inspect the chart. Great. Maybe I don't even need to go through all of that if I just refresh this page. Yep, the chart's

already there. Look at how cool this is. Okay, so we're done. We're going to skip past all those other steps because we don't need to test those. Everything works. Let's keep going. Okay, so next we're going to add the ability to actually publish the test page. So let's do that. So this will allow me to share the test results with anybody who wants to see it. And as a reminder, if you ever wanted to go back and start from a previous step or you ever wanted to add more functionality, you can easily do both with Pythagora. So here we go, Code Monkey writing more code, and now you can see we are almost done, and we are on the fifth of six epics. Okay, so now it's going to tell us to test it. All right, let's see what we got to do. We know all these steps. Let's go ahead and click Start right away, click on the test name, and then click Publish. Okay, so we are here. Let's click into one of them, and we got another error. There was an error.

serving your request. Okay, so let's try to fix it. So when I click on a view test details page, I get this error. All right, so of course, it's going to ask me to reproduce it. It's going to add logging, and let's wait for it to finish that. And to me, the debugging is actually the coolest part, just describing what the issue is, reproducing it, adding a bunch of logs, and then it figures it out and fixes it. It's been pretty seamless so far. Okay, so we're going to start up the app. I'm going to refresh this page. Still getting the same error. Perfect. Copy back and logs, and then click Send. We don't need front-end logs, and I don't need any additional feedback. And we can clearly see where the problem is. I mean, that's amazing. Now it's going to attempt to fix it. We're going to test it again, make sure it works, and then we're going to continue. So to reproduce it, let's do it. So start the app, refresh the page, and

It works okay, and there's the published button, but before we test that, let's go back and just say the issue was fixed. Now we will see the publish button. Let's go ahead and publish it. An error occurred while publishing the test. Okay, so let's copy that. I'm going to refresh the page. Oops, oh right, I have to start the app again. Let's refresh the page. Let's click publish test. An error occurred while publishing the test. So, I'm going to copy that. I'm going to refresh the page, and it does say published, interestingly enough. So, I think there's a bug, but let let's tell it exactly what happened. All right, so it seems to have worked, but the popup after clicking publish said an error occurred. Okay, so here it is. Start the app, refresh the page. This one's already published, so let's go back to the test page. I'm going to click another one, go down to the bottom, click publish test. An error occurred. Okay, so it's still there, and I'll refresh the page, and it says status.

publish, so it did work. Same thing, copy back-end logs and continue. No front-end logs, no feedback needed. All right, so it seems like it found the issue. Oh no, it's adding more logs. Okay, so it needs a little bit more information. So this is the first time we've actually needed to do this twice. So this is going to be interesting. Okay, so let's recreate it again. Start the app, refresh the page. We're going to need a new one. So let's go back to test, go to 44, publish, an error occurred, click okay, go back, copy back-end logs, send it. Don't need any front-end logs, don't need an additional feedback. Okay, we can see that the back-end is functioning properly. So we did know that. That's good. And now it's adding more logs to identify why it's still getting an error. So everything's working, it's just telling us the wrong thing. I need human intervention. Open the browser's developer tools and right-click and hit inspect. Okay, console, and there it is. So we see some issues.

here all right so I'm going to copy all of those front-end logs. Provide any logs or errors. Okay, continue. So, but it says copy backend log. So, let's redo it again. Test. Let's go down to two of two publish. There it is. All right. Now I'm going to inspect console. There it is. Okay. So, there's that error. Let's see if there's any more info in there. Yeah. So, we'll get all of that info. Copy backend logs. Continue. This time we do have front-end logs. Then let's hit send and then continue without feedback. Okay. Now we can clearly see. All right. Perfect. Now it is fixing the bug. All right. Reproduce the issue. Great. Let's try it out again. We're going to start the app. Refresh the page. Here's one of one. Let's publish the test. And it worked. Perfect. All right. So, yes, this fixed the issue. Yes, it says publish. Now everything works. Beautiful. Next, implement the backend logic of publishing. Yes, go ahead. So, we are going to click the publish button. The publish test button should be removed. Log out.

and then visit, and then we should see that published test. All right, so even though it says to check that, if we just look at the progress bar, that's actually going to be the final epic, is actually making the homepage display all that information. So for now, we just click everything works. Now we are on the final step, update the homepage to display the list of published tests. And again, this means we're not logged in, but we're able to see the test, thus they are published. All right, so that should be done determining how to test it. It's going to give me all the steps, but we don't need to follow all the steps. Let's just start the app and just see if we have a bunch of tests. So I'm going to log out, go to the homepage. There they are. Boom, look at that. Let's click into one of them. Ah, so it won't show us the actual test results. All right, so even though it does give me an issue, if I look at the progress, I can see that's

the very next thing that we're going to be doing, so I'm just going to say everything's working and let's continue. And then next, what we should be able to do is reload the test View page that has been published as a non-authenticated user, and we should be able to see it. So yeah, sometimes what will happen is it tells you to test for something that it hasn't written yet, and that's okay. Just keep an eye on the progress bar so you can see what's coming, and if you see that that functionality is coming, go ahead and just continue. Everything's working. All right, let's give it a try. So let's start the app, come back here, let's refresh the page, and there are the details. I'm not logged in. We can see that because it says log in right there, and we have all of the results. Perfect. Okay, let's continue. Everything works. Now it's creating the read me. Awesome. Tells you exactly how to set it all up, how to reproduce all of these things that we just did. All right, so it

says next add navigation links, but we already have that. I tested it, so we don't need it anymore. So, let's skip. And then the last step, are you satisfied? So, it's asking us, yep, app works great. Let's submit now. Let's deploy it. Here's the tab for deploying. Want to deploy your project? One click. Go ahead and deploy. Look how easy this is. All right. And as it's deploying, and it just finished, I will give you this link. I'll drop it in the description below, and you can try it. You can try this application if you want. So, click open URL. Let's open it. And there it is, deployed live on the web. Very easy, awesome. And all of the data is still there, which is really cool. So, that's it. We're done. I just created a full-stack application, a very valuable one, one that runs benchmark tests against different LLMs. It has authentication, it has admin, it has publishing tests to the web with non-authenticated users, hundreds, maybe even thousands of lines of code, and I wrote absolutely zero of them. And in

fact. If I click this little menu right here, I can see exactly how many lines of code were written: 1,637 different files, and again, I wrote none of them. So, very cool, amazing work by Pythagora. Thank you again to Pythagora for working with me on this video. This has been awesome. I know this is on the longer side of the video, but I wanted to walk you through what it looks like to actually build a full-stack application. Now, if I did want to continue and build that more functionality, I could easily do that as well. Thanks again to Pythagora. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

This AI Coder Is On Another Level (Pythagora Tutorial)

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

이 동영상은 2시간 이내에 1,600줄의 코드로 Pythagora를 사용하여 풀스택 LLM 벤치마킹 애플리케이션을 구축하는 방법을 보여줍니다.

이 애플리케이션은 사용자 인증, 관리자 대시보드, 테스트 생성 및 게시 기능을 갖춘 일련의 테스트 문제에 대해 다양한 대규모 언어 모델을 테스트하는 것을 목표로 합니다.

Pythagora의 반복적인 접근 방식은 다양한 개발 역할을 처리하는 여러 에이전트를 포함하므로 프로세스를 효율적이고 협업적으로 만듭니다.

사용자 모델 업데이트 및 관리자 사용자 만들기

기본값이 뷰어인 역할 필드를 포함하도록 사용자 모델이 업데이트되고 있습니다.

개발자 에이전트가 진행률 표시줄과 추적용 로그와 함께 새 코드를 작성하고 표시하고 있습니다.

사용자를 등록하고, 데이터베이스 항목을 확인하고, 스크립트를 통해 관리자 사용자를 생성하여 애플리케이션을 테스트하고 있습니다.

Code Monkey로 실제 애플리케이션 구축하기

코드 몽키 에이전트가 프로세스의 각 단계를 설명하면서 실시간으로 코드를 업데이트하고 있습니다.

빌드 중인 애플리케이션에는 사용자 역할 관리, 테스트 데이터 모집단, 테스트 항목 삭제 버튼이 있는 관리자 대시보드가 포함되어 있습니다.

워크플로가 원활하고 효율적이어서 작업을 빠르게 진행하고 테스트할 수 있으며, 궁극적으로 종합적인 실제 애플리케이션을 만들 수 있습니다.

테스트 및 디버깅 테스트 생성 기능

테스트를 채우는 스크립트가 처음에는 테스트를 추가하는 대신 5개의 테스트를 다시 생성하여 문제를 일으켰습니다.

기존 테스트를 삭제하지 않고 새 테스트 5개를 추가하도록 코드를 업데이트하여 문제를 해결했습니다.

테스트 생성 양식은 성공적으로 구현되었지만 버그로 인해 테스트 이름이 삭제되어 문제를 파악하기 위해 추가 디버깅 및 로깅이 필요했습니다.

디버깅 및 테스트 테스트 실행 흐름

처음에는 샘플 데이터에 이름 필드가 포함되어 있지 않아 테스트 이름이 누락되었지만 앱 시작 후 페이지를 새로 고치면 문제가 해결되었습니다.

테스트 이름을 입력하도록 테스트 생성 양식을 업데이트하고 테스트 실행을 위한 백엔드 로직을 구현했습니다.

테스트 실행을 위한 실시간 진행률 추적을 테스트한 결과, 여러 LLM을 사용할 때 진행률 표시줄이 나타나지만 코드가 LLM을 효과적으로 활용하고 있는지 불분명했습니다.

테스트 실행이 성공적으로 이루어지고 있으며 진행 상황 업데이트와 상세 로그가 추가되고 있습니다.

테스트 결과를 데이터베이스에 저장하는 문제가 해결되어 Anthropics를 사용한 평가 프로세스가 구현되고 있습니다.

그러나 테스트 페이지에서 오류가 발생했는데, 이를 쉽게 찾아내 해결하여 테스트가 성공적으로 실행되고 정확한 결과가 표시되고 있습니다.

테스트 페이지 애플리케이션 빌드 및 디버깅

테스트 보기 페이지 및 막대 차트와 같은 기능이 구현되는 등 애플리케이션이 순조롭게 진행되고 있습니다.

코드 원숭이는 더 나은 기능을 위해 스스로 반영하고 코드를 다시 작성하고 있습니다.

애플리케이션이 실시간으로 오류를 식별하고 수정하는 등 디버깅은 원활하게 진행되고 있지만 게시 버튼과 관련된 지속적인 문제로 인해 추가 로깅과 사람의 개입이 필요해 해결을 위해 노력하고 있습니다.

Pythagora로 풀스택 애플리케이션 구축하기

이 프로세스에는 백엔드 및 프론트엔드 로그를 복사하여 문제를 해결하고 버그를 수정하여 성공적으로 테스트를 게시하는 과정이 포함되었습니다.

그런 다음 애플리케이션은 게시를 위한 백엔드 로직을 구현하고 게시된 테스트를 표시하도록 홈페이지를 업데이트하여 인증되지 않은 사용자도 볼 수 있도록 했습니다.

마지막으로 이 프로젝트는 총 1,637줄의 코드만 작성하여 웹에 손쉽게 라이브 배포되어 풀스택 애플리케이션 구축의 용이성을 보여주었습니다.