대규모 언어 모델 에이전트: 데이터 분석의 혁신

대규모 언어 모델 에이전트는 자동화 및 데이터 분석을 강화하여 사용자가 자율적인 추론을 통해 여러 테이블과 데이터베이스를 쿼리할 수 있도록 지원합니다.

### Prolego의 AI 전략 시리즈: 통합 NLQ를 위한 대규모 언어 모델 에이전트의 힘 활용하기
* 🤖 대규모 언어 모델 에이전트는 추론하고 자율적으로 작업을 실행할 수 있는 소프트웨어 엔티티로, 특히 **여러 테이블을 다룰 때 데이터베이스 쿼리 문제를 해결**하는 데 중요한 역할을 합니다.
* 🔍 관계형 데이터베이스는 인간의 사고 및 의사소통과 자연스럽게 일치하지 않기 때문에 **사람조차도 데이터에서 데이터를 추출하기가 어렵습니다.**
* 📈 대규모 언어 모델 에이전트로 구동되는 통합 자연어 쿼리(NLQ)는 사용자가 일상 언어로 질문하고 여러 소스에서 체계적으로 데이터를 수집하여 답변을 받을 수 있으므로 엔터프라이즈 AI의 **'킬러 앱'**으로 꼽힙니다.
* 🛠 상담원 솔루션을 구축하면서 얻은 주요 교훈 중 하나는 **상담원이 서로 다른 테이블이 어떻게 연결되어 있는지 이해할 수 있도록 데이터베이스 구조에 대한 메타데이터**를 제공하는 것이 중요하다는 것입니다.
* 🧠 필터링 메커니즘으로 **"합리성 테스트"**를 구현하면 상담원이 관련 없는 질문에 답변을 시도하는 것을 방지하는 데 도움이 됩니다.
* 🚀 이러한 유형의 솔루션을 구현하기 위해 프로젝트를 시작할 때는 **작고 가치 있는 데이터베이스**로 시작하여 점차 복잡성을 추가하는 것이 좋습니다.
* 📊 현재 이 팀은 여러 데이터베이스 및 서로 다른 데이터 원본과 관련된 문제를 해결하기 위해 기획 에이전트가 여러 데이터 원본의 쿼리를 담당하는 **여러 독립 작업자와 협력**하는 접근 방식을 고려하고 있습니다.

large language model agents will supercharge your automation and data analytics, and every product manager and analytics leader needs to know about them. By the end of this video, you'll understand what agents can do and how to begin deploying them at your company. [Music] Welcome to episode 4 in Prolego's AI Strategy Series. In episode 3, I covered Unified Natural Language Query, the killer app for Enterprise AI. I demonstrated Unified NLQ against a single data table and explained why it is undeniably the future of business intelligence. Well, today we're going to continue our Unified NLQ journey by introducing multiple tables. Solving this critical challenge requires the use of large language model agents, so let's get started by talking about them. If you find the term software agent confusing, you're in good company. Most of the definitions are too abstract, and the terms mean different things in different situations. So let's come up with a simpler definition that suits our purposes. A large language model agent is a software entity capable of reasoning.

and autonomously executing tasks. We're going to spend a lot of time digging into agents in future episodes, but today I'll constrain the discussion to querying databases. But before we go any further, let's discuss why getting data out of a database is such a difficult problem. Relational databases were designed to overcome the challenges of maintaining data consistency and integrity. Unfortunately, this design isn't naturally aligned with how most people think and communicate. In fact, every developer knows the challenge of creating communication interfaces between software and databases. So getting data out of multiple tables in a database is a difficult task even for people, but agents can do it as you're about to see. Here is another demo of unified NLQ. First, let's take a look at the data we're going to be using in this episode. There are eight tables representing information in a bank's CRM system, such as customers, credit scores, and accounts. Once again, we want to empower users to begin getting answers about this data by asking simple questions, but this is significantly harder than getting data from one table.

as I demonstrated in episode three asking questions across many tables requires an agent to come up with a plan. Here is how the demo works: The user asks a question using natural language, like how many products does the person who most recently opened a mortgage have. The answer will appear at the bottom, and in the middle is a status window telling the user what is happening. And it worked! Let's take a look at the message log below to see what happened. The chatbot message log shows the dialogue between this application, which is the orangish color, and the agent, which is OpenAI's GPT-4 model in gray. The workflow begins with the agent evaluating the feasibility of answering the question. In this case, it found the question to be reasonable, and the process can continue. So you're probably wondering, what is an unreasonable question? Well, an example would be asking how to insert Charlie into the database. Not surprisingly, the agent advises that it cannot help insert Charlie into the database. Thanks for your help, Charlie. Now, I realize this is a silly example of an unreasonable question, but as we'll

discuss with Justin in a moment. One of the keys to making a functioning NLQ application is creating the interfaces that constrain and guide the users and the agent to the right answers. Evaluating question reasonableness is a simple way to filter out the obviously infeasible questions. Without it, the agent can wander down random directions, as can be seen in the creative open source projects like Auto GPT. After evaluating the question for reasonableness, the agent creates a plan for answering the question. The agent needs to find the most recently open mortgage, get the customer's ID, and then use the ID to count the number of products the customer has. This is an excellent plan that unfortunately just won't work. The mortgage table doesn't have a customer ID field. Who would design a mortgage table without a customer ID? Well, we deliberately made this example more complex for the agent because this is very typical of what happens in the real world. As you know, data is messy. Unfortunately, the question cannot be answered without the customer ID. What happens next is truly amazing, and it is

why unified NLQ is the killer app for Enterprise AI. When our application attempted to run the SQL from the agent, it detected the error and relayed the problem back to the agent. The agent recovers, reasons through the problem, noting that it can use the account number from the mortgage table to look up the customer's ID in the products table. So, it gets the account number from the most recently opened mortgage. Using that account number, it gets the correct customer ID. And from here, it is home free. Now let's see if we can push the agent a bit harder. I decided to repeat the question and also ask, What is their name and credit score? The agent had no problem extending the search with a few extra queries. So, I decided to push it harder still. This time, I also told it to calculate net worth, and it replied with a wealth of helpful information, including what it could do, could not do, and alternative options. For example, it guides me to calculating the balance in multiple accounts. There is a lot going on here and the transformative impact of

this technology may not be obvious to everyone at first glance, so let me do a quick summary of where we've come in the past two episodes. In episode 3, I showed how you can create applications that allow your customers to get answers by asking questions using everyday language. I also showed you how the large language models could provide your customers with valuable context about the data, and today I demonstrated how you can deploy the agents to answer questions by systematically gathering data from multiple data sources and recovering from mistakes when they make them. Unified NLQ will be the killer app for Enterprise AI because the demand for this capability will drive adoption across your company. I have yet to encounter a single executive at one of Prolego's clients who wasn't looking for this type of insight from their data. And let's be honest, the projects that create instant value for executives are the ones most likely to get funded. But this is a canned demo on a tiny fictitious database. We took a big leap from episode 2 to episode 3, but we have yet to

address challenges like multiple databases, document repositories, tools, and APIs. Fortunately, Pro Lego is already working through these challenges and will cover them in upcoming episodes. I sincerely hope we can continue this exciting journey together and that you'll stay in touch so I can email you when I release new episodes. The link to our newsletter is down in the show notes, and you can hit subscribe on YouTube to be alerted as soon as we release a new episode. Let's now jump into the best part of the episode and get some deeper insights from Justin Pounders, who led the development of this solution. Hello, Justin. Hi, Kevin. So for the benefit of our audience, could you describe basically how the application works at a high level, and especially how the agent works? In short, we want to ask potentially a multi-part question and have the agent be smart enough to break down that question into multiple parts, run multiple queries, and then synthesize an answer. What were some of the key learnings from building the agent solution? Well, I think two

things come to mind. First, in this case, we have a database consisting of multiple tables, and the tables are connected using, you know, foreign keys, for example. So, one of the things we have to do up front is if the agent is going to know how to look at this database and understand the structure and what pieces can be joined to what other pieces. Well, we have to tell the agent how that database was constructed. And so, that's the first thing that we need to put into the context. Not just, Here's a bunch of data, but, This column and this table can be linked or joined to this column in this table, so that the agent understands how it can go across, traverse the database structure intelligently. The second big thing, we need to lay out the format for the interaction between us or our software and the LLM agent. So, for example, and this is something that was a big learning in this version of the agent because we started off by giving it very strict instructions, say, Issue a thought and call it a thought, and then issue an.

action which should be a function call, and then you know put your final answer like this. And so we gave it a very uh specific format for how to talk to us via the API, and it turns out that didn't work very well. Uh Chad CPT was trained to be a dialogue agent, and so we said, well hey, rather than trying to you know tell it specifically how to talk to us, let's just have a conversation and have the right code to understand what to do with those different pieces of the conversation. And turns out that worked a lot better. I guess to your previous comment, metadata about the tables is key to getting the llm to understand what's happening. So you just passing it, you know, column names without other information probably isn't going to work very well. No, and especially not in this case where we have a full database. I think there's eight different tables, and certain tables can be joined with other tables on certain Fields. So we had to be very explicit in that and the way we did that.

was actually introspecting the in this case a sqlite database, and we can pull out the SQL creation statements. So, the statements that were used to create the tables, and in those statements, we specifically say what's primary key and what's a foreign key, and how those references and links work. And so, even though it's just SQL that we're dumping into the context, the agent, the LLM chat GPT in this case, is smart enough to look at those SQL creation statements and understand, okay, I can join this table with this table on these attributes, these column names, to pull together the appropriate SQL query and then generate the final response, which is pretty cool to see. Yeah, wow, it's amazing. Every time, I'm running out of hyperboles after a while, but it is truly amazing. Um, and that's not obvious, by the way, right, because we could just use language to describe the database, but it turns out it's more, I think, specific and concise. I mean, all of them understand SQL, so let's just give it the sequel and

let it work with that it's com. I would have never expected it. I would have expected you have to actually describe each one of the The Columns using natural language. Yeah, so I guess in moving on in the beginning you start with a like a filtering up front which is the reasonableness test like why did you do this and how did you implement it. Yeah, so why we implemented it was the agent that interacts with the database. It's I view it as this narrow tool so it's a it's an agent that's responsible for given the context of the problem and the description of the database interacting with the data in that database via SQL and so it's very sort of constrained it's it's really good at doing that one thing but what if I ask it something that's not relevant to the database so in this case the database had Bank customers and products and accounts and things like that but if I ask about account opening forms or if I ask about transactions that aren't in the database well this agent wants

really hard to solve my problem, and so it's going to start trying to squeeze some information out of the database, or making up answers, or giving me suggestions. You know, which none of those things are necessarily wrong, but it's not going to be able to answer the question. And so we found it's much more efficient to put a screening question up front. Say, Hey, here's the question. Is this appropriate? Is this a reasonable question given the data that you have access to, in the context that you're aware of? And it turns out, if you ask the agent, You know, how do I make a ham sandwich? He'll come back immediately, say, This is not a reasonable question. I can't answer it. So we can stop that line of calls in that line of reasoning right there. Or if you ask, How to put a dog inside a database? It doesn't feel like that it's a very reasonable question either. Let's just say a client comes along and wants to start a project where they can begin implementing this type of solution in their environment. How would

you start thinking about like Define the problem, constraining it, and breaking it down into something that you can start executing on, and systematically isolating risk. I would start small in the sense that don't throw it at a whole data lake with a bunch of document scores and databases of different kinds of structure. That's not a good place to start. I wouldn't start there, no. Maybe just one database, and maybe this is a valuable data database because people interact with it frequently. Maybe there's a lot of value, but it's one system, so you could describe that one schema and then just implement this type of agent-based question answering over that single system, over that single database. Then you can figure out what prompting techniques work. You can add guardrails or sort of constraints like the check for reasonableness. These little knobs and dials, you can kind of get that all figured out for this constrained problem. And then along the way, as you see people interact with the database using this

agent, you could look for patterns so most people are asking these sorts of questions versus those sorts of questions. And so you could further tweak and refine your prompt. Awesome, all right, and final question is what are you and the team working on now? What can we expect to see in upcoming episodes? What we're really thinking a lot about right now is moving away you're adding complexity. So we started with one table, then we went to one database. Okay, well, what if we now have two, three, four, five databases and we want an agent to be able to pull information and reason over these multiple disparate data sources? How do we do that? And some of the ideas we're throwing around now is rather than having a single agent try to understand what's in all these different data sources and handle all the coordination between these data calls, that's probably not going to work well. You're going to end up with a ton of information in your context. So rather than taking that kind of One agent to do everything, I think we're going to have more success with

having an agent that's responsible for planning, and then that agent can spin off different workers. So worker one goes to this database, worker two goes to this database, and then we pull together the results from these sort of independent workers to keep scope and context Limited. This was fantastic information, thank you so much for sharing, and uh, can't wait to see what comes next. Cool, thanks Kevin. Well, that's a wrap in episode four. If you're interested in developing a unified nlq capability at your company, and of course you are, then why not hire Pro Lego to help accelerate your projects and turn your people into AI experts. Hey Russ, how can someone learn about Pro Lego Services? Just shoot me an email at Russ@ProLego.com. So what are you waiting for? Have a great day. [Music]

Ep. 4 - Supercharge Data Analytics with LLM Agents

We are an elite team of AI engineers and creative technologists committed to guiding the world’s largest companies through their AI transformation.

We do this by building custom, innovative solutions to solve complex business problems. We also help drive the cultural and process changes necessary to fully leverage AI.


Prolego

데이터 분석에서 대규모 언어 모델 에이전트의 힘

대규모 언어 모델 에이전트는 자동화 및 데이터 분석에 매우 중요하며, 추론과 자율적인 작업 실행을 제공합니다.

이러한 에이전트는 특히 여러 테이블을 다룰 때 데이터베이스 쿼리에 유용하며, 데이터베이스 설계와 인간 커뮤니케이션을 일치시켜야 하는 과제를 극복할 수 있습니다.

대규모 언어 모델 에이전트가 포함된 통합 NLQ의 데모는 자연어 질문을 기반으로 답변을 제공하고 계획을 실행하여 궁극적으로 데이터 분석 프로세스를 강화할 수 있는 기능을 보여줍니다.

NLQ 애플리케이션에서 질문의 합리성 평가의 중요성

질문의 타당성을 평가하는 것은 상담원에게 올바른 답변을 안내하고 실현 불가능한 질문을 걸러내는 데 매우 중요합니다.

실제 시나리오에서는 지저분한 데이터와 예기치 않은 오류를 처리하는 것이 일반적이며, 상담원이 계획을 복구하고 조정해야 하는 경우가 많습니다.

통합 NLQ는 데이터로부터 가치 있는 인사이트를 제공할 수 있는 혁신적인 잠재력을 가지고 있으며, 비즈니스 의사 결정 및 프로젝트 자금 조달에 즉각적인 가치를 제공하기 때문에 경영진 사이에서 높은 수요를 보이고 있습니다.

프로 레고 솔루션: 데이터 과제 극복하기

프로 레고는 다음 에피소드에서 여러 데이터베이스, 문서 저장소, 도구 및 API의 문제를 다룰 예정입니다.

상담원 솔루션을 구축하면서 얻은 주요 교훈은 상담원이 데이터베이스 구조를 이해하도록 도와야 한다는 점과 API를 통한 상호 작용에 대한 엄격한 지침보다는 유연한 대화 형식의 중요성이라는 점입니다.

특히 전체 데이터베이스와 여러 테이블을 다룰 때는 테이블에 대한 메타데이터를 제공하고 테이블을 조인하는 방법을 명시적으로 설명하는 것이 중요합니다.

데이터베이스 상호 작용에서 SQL의 역할 이해

SQL 생성 문은 기본 키, 외래 키 및 테이블 관계를 지정하는 데 사용되어 LLM 채팅 GPT가 적절한 SQL 쿼리를 생성하는 방법을 이해하도록 합니다.

LLM 채팅 GPT는 질문이 기존 데이터베이스 콘텐츠와 직접 관련이 없는 경우에도 SQL 문을 사용하여 데이터베이스의 데이터와 상호 작용하는 데 능숙합니다.

SQL을 사용하면 상담원이 데이터베이스와 상호작용할 수 있는 구체적이고 간결한 방법을 제공하므로 효율적이고 효과적인 응답을 할 수 있습니다.

데이터베이스 질문 답변의 효율성 향상

상호작용의 시작 부분에 선별 질문을 넣으면 불합리한 질문을 걸러내고 시간을 절약하는 데 도움이 됩니다.

하나의 데이터베이스로 작게 시작하여 상담원 기반 질문 답변 시스템을 구현하면 위험을 체계적으로 격리할 수 있습니다.

그 과정에서 가드레일과 제약 조건을 추가하면 보다 효율적인 데이터베이스 질문 답변 솔루션을 위해 프로세스를 미세 조정하는 데 도움이 됩니다.

다중 데이터베이스 통합으로 초점 전환

현재 이 팀은 여러 이질적인 데이터 소스를 통합하고 추론하는 과제에 집중하고 있습니다.

이들은 범위와 컨텍스트를 제한하는 것을 목표로 기획 에이전트가 각 데이터베이스에 대해 독립적인 작업자를 조율하는 아이디어를 모색하고 있습니다.

다음 에피소드에서는 복잡한 데이터 통합을 처리하기 위해 단일 에이전트에서 다중 작업자 접근 방식으로의 전환에 대해 다룰 예정입니다.