AI를 통한 법률 업무의 혁신: 판례 텍스트의 여정

Jake Heller는 AI를 활용하여 법률 테크에 혁명을 일으켜 Co-Counsel을 설립한 후 6억 5,000만 달러의 투자금을 유치했습니다.

### 리걸 테크의 혁신: 판례 텍스트와 AI 통합의 여정
* 🚀 Case Text의 창립자인 Jake Heller는 **GPT-4 출시 후 불과 두 달 만에 기업 가치가 0달러에서 6억 5천만 달러로 상승**한 자신의 여정을 달에 처음 착륙한 사람의 여정에 비유합니다.
* 💻 원래 법률 기술에 중점을 두었던 이 회사는 **GPT-4의 기능을 기반으로 한 획기적인 AI 법률 비서인 Co-Counsel**을 개발하는 방향으로 전환하여 변호사들이 이전에는 며칠이 걸리던 복잡한 작업을 자동화할 수 있도록 했습니다.
* 📈 초창기에는 변호사들의 높은 시간 가치로 인해 변호사로부터 사용자 제작 콘텐츠를 만들려는 시도가 실패로 돌아갔고, 대신 자연어 처리에 많은 투자를 하게 되면서 **Case Text는 주목을 받는 데 어려움을 겪음**.
* 🛠 헬러는 법률 분야는 높은 정확도와 정밀도를 요구하며 사소한 오류도 중대한 결과를 초래하기 때문에 **신뢰할 수 있는 AI 도구를 만드는 데 있어 테스트 중심의 개발 접근 방식**이 중요하다고 강조했습니다.
* 💡 변호사들이 업무에서 AI의 혁신적 잠재력을 빠르게 인식하고 법률 커뮤니티의 수요와 관심이 급증하면서 **Co-Counsel의 출시는 전환점**이 되었습니다.
* 🔍 헬러의 인사이트에 따르면 개발 과정에서 복잡한 법률 업무를 관리 가능한 단계로 세분화하여 **AI가 법률 전문가의 엄격한 기준**을 충족하는 수준의 성능을 발휘할 수 있도록 했다는 사실이 밝혀졌습니다.
* ⚖ OpenAI의 새로운 모델인 O1의 도입으로 언어 모델의 추론 및 실행 기능 기능이 향상되어 **미션 크리티컬 애플리케이션에서의 활용도**가 더욱 향상될 것으로 기대되는 만큼 AI가 더욱 발전할 가능성이 있습니다.
* 🌟 헬러는 다양한 분야의 혁신가들이 AI의 잠재력을 수용하도록 장려하며, 도전 과제는 존재하지만 **가치 있는 AI 애플리케이션을 만드는 길은 분명하고 달성 가능**하다고 강조합니다.

this is their first-ever experience talking to this God-like feeling, you know, AI that was all of a sudden doing these tasks that would take me when I practice like a whole day, and it's being done in a minute and a half. The whole company, all 120 of us, did not sleep for those, you know, months before GBD4. We felt like we had this amazing opportunity to run far ahead of the market. That's why you're the first man on the moon. [Music] Yeah, welcome back to another episode of the Light Cone. I'm Gary, this is Jared, and Diana. Harge is out, but he'll be back on the next one. And today, we have a very special guest, Jake Heller of Case Text. I think of Jake as a little bit like one of the first people on the surface of the Moon. He created Case Text more than, I think, 11, 12 years ago, actually. And in the first 10 years, he went from $0 to $100 million valuation. And then, in a matter of 2 months after the release of

GPT-4, that valuation went to a liquid exit to Thompson Reuters for $650 million. So, you have a lot of lessons about how to create real value from really large language models. I think you were, you know, our friends in YC, one of the first people to actually realize this is a sea change and revolution. And not only that, we're going to bet the company on it, and you were super right. So, welcome, Jake. Happy to be here. One of the cool things, I think, about Jake's story and the reason why we wanted to bring him on today is that if you just look at the companies that good founders are starting now, it's a lot of vertical AI agents. I mean, I was trying to count the ones in S24. We have literally dozens of the YC companies in the last batch for building vertical-specific AI agents. And I think Jake is the founder who is currently running the most successful vertical AI agent. It's by far the largest acquisition, and it's actually deployed at scale in a lot of

mission-critical situations, and the inspiration for this was, uh, we hosted this retreat a few months ago, and Jake gave an incredible talk about how he built it. We thought that it'd be super useful for people who watch the light cone, who are interested in this area, to hear directly from one of the most successful builders in this area how he did it. So, how did you do it? Well, first of all, like a lot of these things, um, there's a certain amount of luck. Over the course of our decade-long uh journey, we started investing very deeply in AI uh and natural language processing, and we became close with a number of different research labs, including some of the folks at OpenAI. And when it came time for them to start testing early versions, uh, we didn't realize it was GPT-4 at the time, but what was GPT-4? We got a very early kind of like view of it, and so, you know, months before the public release of GPT-4, you know, we as a company

were all under NDA, all working on this thing, and I'll never forget the first time I saw it. It took maybe 48 hours for us to decide to take every single person of the company and shift what they were working on, from what the projects we were then working on at the time, to 100% of the company, all working on building this new product we call co-counsel, based on the GPT-4 technology. How many people was that? We're about 20 people at the time. So, you took like 120 people and completely changed what they were all working on? Yes, yes, yes, in 48 hours. Yes, and for the people watching, uh, Casex originally, I mean, had always been in the legal space. You're a lawyer, and you built something for yourself, and you know, sort of the first versions of it were actually sort of, uh, annotated versions of case law, actually. Yeah, that's exactly right. So, in the very early origins of the company, the mission of the company, what we're always focused on, is how can we build something that

brings the best of technology to the legal space. Um, I, as a lawyer, I actually like the job a lot. The parts of my job that I hated the most was when I had to interact with the technology that lawyers have to use um regularly to get the job done. I remember thinking, and this is like 2012 when I was at a law firm, if I want want to do something really trivial, I had like a new iPhone at the time. I can go on Google and find like movie times or where's the closest open Thai restaurant with vegetarian options. That was super easy. But if I wanted to find the piece of evidence that was going to exonerate my client and and make it so he doesn't have to go to jail for the rest of his life or the um key legal case that will help me win a billion dollar lawsuit, well, that's going to be like 5 days in a row till 5:00 a.m. every day. It's like there's got to be a better way. What is the

process as a lawyer. You would have to read the stacks and stacks of documents. Pretty much, yeah. Um, right before I started practicing, before everything went virtual or like online, uh, you would literally be in a basement with bankers' boxes full of documents, reading them one by one by one to try to find, you know, all the emails in a company like Fiser or Google to see if there was potential fraud or um, and then if you wanted to find case law slightly before my time, you'd literally go to the library and open up books and just start reading, and you know, new products were coming out that were some of the first web-based research tools, but they were pretty clunky. It was just hard to find the relevant information. You couldn't do Ctrl F or any of this stuff, basically, basally not, yeah. And what was interesting about your is you also happened to be the rare breed of having also computer science training, so this must have driven you nuts. Yeah, exactly. I mean, in the law firm, I'll

never forget. I was building browser plugins to go on top of the tools I was using just to make my life more efficient and effective. And actually, one of the reasons I left the law firm to start a company and apply to YC was I got in trouble with the general counsel who thought, like, Hey, why are you spending all your time, you know, doing this tech stuff? And also made at the time very clear that my law firm owns all that technology. So, I decided to do something different. So, do you want to tell us a little bit about the first 10 years of case text, the sort of long slog in the pre-LLM era? One of the lessons here, I think, that I took away from that time period is that when you start a company, you may not get the exact right, you may have the right kind of general direction, you know, there's a problem you're trying to solve, but it could take a very long time to figure out what the solution is. For us, for

example you know we saw that there was this kind of combined issue of like bad technology in the legal sphere but also like this very like a lot of lawyers use content to do things like research and understand like what the law is and so we thought okay well we can do the technology better but how are we going get this content and we spent like a couple of years trying to get as Gary said lawyers to annotate case law and to provide information so it's like a ugc site like user generated that was a big focus of ours like the kind of onew punch of better technology but also better content um we you know at the time our heroes were like stock overflow and Wikipedia and GitHub and other kind of Open Source or ugc kind of websites and it was a total failure like we could not get lawyers to contribute their time and information and and I think these are just different populations the typical Wikipedia editor has more time on their hands than they know what to do with and so they're adding not all but

but many do, and they're adding content for free, um, and altruistically. Lawyers bill by the hour; their time is incredibly valuable. They're always running out of time; they had no time to kind of contribute to some UGC site. So we had to pivot, and we started investing very deeply. At the time, it was not called AI; it was just like natural language processing and machine learning. We saw that, first of all, we didn't need to create all this UGC to replicate some of the best benefits of what our competitors had in these big content databases. Some of it you can basically do even then on a kind of automated basis. And then also, uh, we were starting to create these user experiences that were, you know, a lot better than what our competitors could offer based on then, at the time, what seems kind of quaint, like AI stuff, like, you know, the same recommendation algorithm that powers Pandora and Spotify's recommended music. You can use, they look at basically is how this song relates to that song.

people listen to this, also listen to this, and this, and this, right? Similarly, we looked at okay cases that cite to, you know, other cases. They all reference earlier opinions, you know. They kind of build out this network of citations, and we found ways that we can check a lawyer's work. They'd upload their work so far and be like, Well, everybody who talks about this case talks about this case too, and you missed that. Um, so cool experiences like that. But the truth is, until the very end, until co-counsel, a lot of what we did were, relatively speaking, kind of incremental improvements on the legal workflow. And one of the things that's kind of weird about this is, um, when there's just an incremental improvement, it's actually pretty easy to ignore. A lot of our clients, they never say this literally, but you kind of get this impression. You walk into the room, their office, and you try to pitch them a product, and you say, This is going to change everything about the way you practice, and they go, Well, I make $5

million a year. I don't want nothing to change. This technology, PL, it's not. I do not want to introduce anything that has the opportunity to make my life at all worse, um, or potentially worse, or potentially more efficient because they build by the hour. It was really only after, like, much later when chat GPD came out, you know, the time we were privately and secretly working on GP4. Chat GPT came out, and all of a sudden, every lawyer in America, probably in the world, saw, Oh my God, I don't know exactly how this is going to change my work, but it's going to change it very substantially. Like, they could feel it. And the same, you know, guys and gals were telling us, I make $5 million a year. Why would it change anything about my life? We like, I make $5 million a year. This is going to change something. I need to be ahead of this. The technology itself, and we'll get into a second, really changed what we can build for life employers, but also the market perceptions of what was like what.

was necessary really Chang as well, and for the first time in our 10 years, you know, even before we launched Co council publicly based on GPT 4, they were calling us like, you know, we know you work on AI, we need to get on top of this, what can you know, what can you show us, what can we work on, and I think it's because the change was not incremental anymore, it was like fundamental, and all of a sudden they had to pay attention, they could not ignore it. I guess, uh, the mental model I have for you is there's this concept of the idea maze, you know, the founder goes in the beginning of the Maze, and they're just like feeling around, like actually, uh, in the arena, talking to, you know, customers, learning, like, where are the walls, which path to go, should I go left or right, like, and then, um, as is actually common for startup Founders in the idea maze, you will actually reach a dead end, and then usually you have to Pivot, yeah, and then I think you have a

very interesting story because you were sort of towards the end of maybe like one of the uh you know parts that weren't going to get you all the way to product Market fit but then lm's Dro and then it's like the maze got shaken up yeah and then you are actually much closer to product Market fit than absolutely anyone else that's yeah I it's exactly right that's why you're the first man on the moon yeah yeah I think I think there's there's cly something to that and the thing is you know each time we got progressed through that maze it felt like maybe now we're at product Market fit you know we were making real Revenue before we um launched co- counil and we had real customers and they said really great things about us I keep on thinking about this article written by Mark andreon in like the early 2000s uh I think it's called the only thing that matters and in it he describes the what it feels like to have product Market fit.

he lists things like your servers will go down, you can't hire support people and sales people fast enough, you're going to eat for a year free at bucks, the kind of famous Woodside, you know, Diner where a lot of VCs will take you, the press, and I read that early on in my career, and I was like, okay, well, that's like hyperbolic, but when we launched Co-council, it was literally exactly that. Our servers were going down, we could not hire support people fast enough, we couldn't hire sales people fast enough, I ate a lot of bucks, you know, before we was a really big day if we in the ABA journal or some other, you know, legal-specific publication. We were on CNN and MSNBC, and like, you know, all of a sudden, everything changed, and that's what real product-market fit looks like. I think Mark was even in like 2005 whenever the article came out, exactly right about it looked like in 2023.

you talk about that crazy time because it was only two months from when you launched co-council to getting bought for $650 million. So, like, what happened in those two months? Well, to be clear, the transaction only closed six months after we launched, but it was two months the conversation started. And so, we started building CO-council. And just to kind of background purposes, the idea we came up with, again, like 48 hours, like a weekend after seeing GPT-4, was something that not but kind of still sound crazy today, but it felt crazy at the time, which is this AI Legal Assistant, by which we mean it's like almost like a new member of the firm. You can just talk to it, not unlike how you might talk to something like ChatGPT today, and give it tasks like, I need you to read these a million documents for me and tell me if there's any evidence of fraud happening in this company. And then, within a couple of hours, it's like, I've

read all the documents. Here's what the summary is, or summarize documents, or do legal research and put together a whole memo after researching, you know, hundreds or thousands of cases, answering the lawyer's initial research question, and so in that sense, it was this like really powerful extension of the workforce of these law firms. That was the concept from the beginning, and we made a very early initial version of it. We started because we couldn't, you know, under our agreement with OpenAI, we could not be public about this product, but they did let us extend the NDA to a handful of our customers. So, we started having our customers use it, and so, you know, for months before GPT-4 was launched publicly, we had a number of law firms. They had no idea they were using GPT-4, but they were like seeing something really special, right? This is actually even before ChatGPT. So, this is their first ever experience talking to this God-like feeling, you know, AI that was all of a sudden doing these

tasks that would take me when I practiced like a whole day, and it's being done in a minute and a half, right? And so, as you might imagine, like, it was nuts. I mean, first of all, the whole company, all 120 of us, did not sleep for those, you know, months before gbd4 was, like, publicly launched. Therefore, could publicly launch the product. We felt like we had this amazing opportunity to run far ahead of the market. Something really beautiful happens when everybody's working super, super hard, which is you iterate so quickly past. And actually, I still see some companies out there, they're stuck where we were in the first month of seeing gbd4, right? Um, and I think it's because they're just not, like, as intensely focused and engaged as we were, were able to be during those, like, couple, like, about 6 months or so before the public launch of gbd4. You kind of, uh, to do this transition, you had to shake the company. You kind of went into deep

founder mode because there was a lot of, uh, push back from employees. I like, Oh, this thing was working. Why should we go into throw ourselves into the deep end of AI? and tell us about that founder mode moment for you. And so, first of all, like, this is especially true. You running a business for 10 years because they've seen you wander through that maze and bump into dead ends, and a lot of those folks have been there for, uh, most or all that time watching, you know, me as the founder saying, We're definitely going this direction. It's definitely going to work, and sometimes it doesn't, and you only get so many of those with employees, right? So, this was maybe my last one that I had with some of these folks, and they're like, Here Jake goes again with this crazy new technology and some idea we're gonna invest deeply in, and, yeah, it took some a job to convince people. And if you imagine, like, what some of the different roles are, if you're in the go-to-market role, if

you're if you're selling or marketing a product, and we're making, you know, we're growing 70-80% year-over-year. We're between $15 and $20 million in ARR. Things weren't like terrible, right? Um, that's great. Yeah, we're great. Yeah, we, but like, so they were like, what, why are we even the board, you know, some of the members, like, I get this immediately, some of them had to be persuaded, right? Um, and about the founder mode moment, like, one thing that really worked for me is, uh, I led the way through example. I built the first version of it myself, um, wow, even with a 120-person company with, like, a whole bunch of engineers and lawyers and stuff like, mhm, before that, you like, opened up your, like, IDE and actually built the thing yourself. Oh yeah, and part of it was, uh, the NDA only extended at first to me and my co-founder. That was it. That was a blessing though. It turned out to be like perfect, and even after the NDA got extended a little bit.

We kept it pretty small at first for the first, like, you know, little bit of time. I made my mind within 48 hours. Whole is going to do this, but we actually only told the company, I think, a week and a half afterward. First got access during that week and a half, like we built the very first version, like prototype version of this. And again, I won't, I'll never forget this. The timing is just so funny. Like we saw it on, like, a Friday. We had it all weekend long. We're working with it. And then Monday was an executive offsite where everybody came, all my executives came, and they expected we're going to be talking about how we're going to hit our sales target for the next quarter. How? And it's like, guys, we're talking about none of that. You know, we are talking about something totally different right now. Let me show you something on my laptop. You know, uh, so yeah, I built the first version myself, but going through that process, me

and and and then a handful of other people I think was really helpful, and we also brought in customers early, and that helped convince a lot of people. As soon as like a skeptical sales or marketing or whatever person or even an engineer was on the other line, end of a zoom call, uh, where um, a customer was was reacting to the product in real time and giving us their honest reactions and like seeing the look on their face. Again, you have to imagine it's almost hard to imagine that the world was like pre-chat GPT, but then there, some of these people were seeing that exact idea for the first time, and they were just blown away, and that really changed minds quickly. I mean, we saw people go through like existential crises live, you know, on Zoom calls, like, Oh my God, you see their expression change exactly in all kinds of ways. It's like, What am I going to do? A lot of the very common reaction amongst the senior attorneys we showed it to was like, Well, that got up retire soon, like, you know, I'

have to deal with this, and some of this was, um, really driven by GPD 4 coming out. Like, you had access to three, you had access even to two. I think we had access; we were in a close relationship again with a lot of the labs, but including OpenAI, and they kept on showing us stuff kind of early on in its development, and they're like, Well, can you build something with this for legal? And every time, we're like, No, this sucks. Like, you know, by the time we got to three and 3.5, it was like, Okay, well, this is plausible-sounding English and sounds kind of like a lawyer, so kudos for that. But it is just making stuff up wildly. Like, we just didn't—it's very hard to connect it to a real use case, especially in legal where it's so important that you actually get the facts right. You can't hallucinate, um, you can't even, you know, make the wrong kinds of assumptions, and we had to do a lot of work with those.

earlier models to even get them close to usable and they just weren't really. I mean, like one example along the way is when GPT-3.5 came out. The study was run, and it showed that GPT-3.5 got a 10th percentile on the bar passage, right? So, like, it did better than some people, actually, but the 10% of them, yeah, probably the ones who were just filling out randomly, basically. When we got early access to GPT-4, we're like, Let's run the study again too, and we worked with OpenAI. We're like, We're going to confirm this. This test is not in the training set, and it wasn't. Totally new test to it. And the test we ran, it did better than 90% of the test takers, right? So, it's like a big difference. And also, we started running some tests like, Okay, here's like four or five cases to read, using those cases, write a memo.

responding to this question, and we did a lot of prompt work to get it to essentially just do it accurately, to cite the actual things in the context that we gave it, and not make things up. And we're like, okay, well, this is very different than we saw before, so it's a big moment for us. And honestly, I'm not sure what the mindset was of the researchers we were working with, but it almost felt like by the time we were having that meeting, it felt like one of those other meetings we'd had in the past where we were getting ready to say, like, this is not going to work for legal, keep on trying. And I think they saw us go through maybe some form of the existential crisis on that call that our customers did. We were like, oh wait, this is super, super, super different. I guess, you know, today we have 01, we have, you know, chain of thoughts reasoning. I think a lot of people look at it as it's not merely the text itself, but also the

instructions that lead up to you know the workflow, but you know, way at the beginning, nobody knew any of this stuff. How did you start? You had your sort of tests that you had written for previous versions of the model. They outperformed, but then there's this moment where you say, Okay, well, now it's something, but what do we do next and how do we do it? So, the process that we started with then, it's actually not too dissimilar to what we're doing today. It started with a question of like, Okay, well, what problem are we trying to solve for the user? Right? The user wants to do research, legal research, so, and they want like a memo answering their question with citations to the original source. So, like, that's the end result. And then we're like, Well, how do we go from that end result, like working backwards almost? What would it take to get there? And what ends up happening a lot with the things that we built for coals, we call them skills, which felt very unique in at

the time I think a lot of companies now call their AI capability skills. So when you're building these skills, um, it turns out it usually takes a lot of work to go from like, say, the customer inputs something, say like a set of documents or a question or what have you, to the end result that they're looking for. And the way that we thought about it was how would the best attorney in the world approach this problem. And so in the case of research, for example, the best attorney would, you know, get the request save from a partner and then break that request down into like actual search queries that run against these platforms. And sometimes they use special search syntax. It looks actually proba like SQL almost, right? So like from the English language query, you have to break it down to these different kind of search queries, maybe a dozen different search queries, you being really diligent. And then um, they'd execute the search queries against these databases of law. They come back with say like 100 results each and then they, you know, the most

diligent best attorney would sit down and just read every single one of these results that come back, all the case law, statutes, regulations, and you start to do things like make notes and summarize and kind of compile like an outline of what your response might be like line by line or paragraph by paragraph. Actually, yeah, 100%. And you start like just taking out those insights you're getting from what you're reading, and then finally, based on all that work and all the citations you've gathered, etc., then finally, you put together your research memo. And so we're like, okay, well, each one of those steps along the way, for the vast majority of them, those were impossible to accomplish with previous technology. But now they're prompts. Think step by step. Yeah, think step by step. Yeah, exactly. But we actually broke it down, each, each, you know, so getting to the final result may be a dozen or two dozen different individual prompts, each of which might, by the way, be thinking step by step them.

themselves, but um, and for each of those prompts, you know, as part of this chain of actions you take to get to the final result, we had a very clear sense of what good looks like, and we were able, you know, we had a series, like a battery of tests before, but this got way more intense where we wrote at first maybe a few dozen tests, and then a few hundred, and a few thousand for every single one of those PRS. So, you know, if the job to be done in the very beginning of this research process, for example, is taking the English language query and breaking it down into search queries, we had a very clear sense of what good search queries look like and wrote like a gold standard answers for giving this input. This is what the output looks like, right? And so our prompt engineers, um, and I was one of them at the very beginning, we all just kind of in it together, we're writing these English language prompts to try to, you know, write the test first.

basically and wrote these English language prompts to try to get it so of 1,200 times they got the right answer 1, 1,199 times or what have you so sort of like um test-driven development oh yeah really approach from doing software engineering to to to prompt that's exactly right and and the funny thing is I never really believed in test-driven development before prompting like I was like the code works it doesn't it's fine like you'll see it when you but like with prompting actually think becomes even more important because of the kind of like nature of these llms they might go in Crazy directions unexpectedly and so you know you might very easily add in a set of instructions to solve one problem you're seeing with these sets of tests and then to break something with these sets of tests so so that that exact kind of theory of test development applies you know 10x more I'd say in the world of prompting there's a lot of uh sort of the naysay are saying that a lot

of companies are just building GPT wrappers, and there's not a lot of IP getting built, but it's actually, there's a lot of finesse to how you explain all of this. Like, can you tell us about all of that and how much more there's to be built? Oh yeah, I mean, I think the thing is when you're actually trying to solve a problem for a customer and actually doing the job, in our case, of like what a young associate might do and do it really well, there are many layers of things you have to add in to actually get the job done. And by the time you add that all up, you're not like a GPT wrapper, you're a full application that may include, in our case, proprietary data sets like the law itself and our annotations to the law that we added automatically. It may include connections into customer databases, in our case in legal, they have these very specific legal-specific document management systems. So, connecting into those is like very important. It may include something

as subtle as like how well you OCR and like what OCR programs you use and how you set those up when you're doing that task of you know one of the tasks that the co-counsel does for example is reviewing large sets of documents. Once you start working a lot of documents, you see like stuff there handwritten all over it and they're like tilted in the scan and there's this crazy thing that they do in law where they print four pages on one page to save like room and LR is going to read it directly across but actually goes you know 1, 2, 3, 4. So by the time you've dealt with like all the edge cases frankly uh not even before you hit the the large language model like everything else up to the large language model um there might be dozens of things you built into your application to actually make it work and work well and then you get to the prompting piece and writing out tests and very specific prompts and and the strategy for how you break down you know

uh a big problem into step by step by step kind of thinking um and and how you feed in the information how you format that information the right way um all of that also becomes like you know your IP and it's very hard to replicate very hard to build and therefore very hard to replicate which is all the business logic which is all even all the very successful SAS companies with very specific domain you need very very custom esoteric Niche Integrations like plug into this esoteric law database yeah absolutely two things I think about it all the time it's like basically all SAS for a while was just like a SQL rapper right like if you think about like very successful companies like Salesforce they built that business logic around basically just databases and connections between like tables and a database and sometimes bridging that gap between um something that like either a very technical person can do but most people can't and making accessible or um bridging that gap between that almost works like you can do a lot of cool Demos in chat GPT.

without building a line of code but that almost works and works, you know, 70% of the time, but going to 100% of the time is a very different kind of task. And people will pay $20 a month for the 70% and maybe $500 or $1,000 a month for something that actually works, depending on the use case, right? So, there's a lot of value gained going that last mile or 100 miles, whatever it is. Yeah, can you talk about how you went from 70% to 100%? Because I think the other knock on this technology that we hear a lot is like, Oh, these LMS hallucinate too much, they're not accurate enough for real-world use. But as you said earlier, like the use case that you're working on is a mission-critical use case. There's like a lot at stake if the agent gives bad information to lawyers who are working on important court cases. How did you make it accurate enough for lawyers who are conservative by nature to trust this test-driven

development framework. First of all, it goes a long way because you can start seeing, you know, patterns and why it's making a mistake. And then you add instructions against that pattern, and then sometimes it still doesn't, you know, do the right thing. And then you kind of really ask yourself, Okay, well, was I being super clear in my instructions? Uh, you know, am I including information that doesn't, you know, it doesn't, it shouldn't see or too much or too little information for it to really get the full context? And usually, like, these things are pretty intelligent, and so usually, you can kind of root cause why you're failing certain tests and then build to a place where you're actually passing those tests and just getting it right, you know. And one of the things we learned is if it after passes, frankly, even like 100 tests, the odd that it will do on any random distribution of, like, user inputs, the next 100,000 100% accurately is, like, very high. One of the things that strikes me that is tricky, like many Founders, we

work with are very tempted to just raw dog it. Yeah, there's like no evales, no test-driven, we're just like vibes only, prompt engineering, and maybe. I mean, you switched over to this very quickly then. Like, was it just obvious from the beginning? You're like, we just can't do it that other way. We should not raw dog any of these prompts. Yeah, I think the biggest thing, first of all, depends on the use case. For a lot of things that we were working on, for better or for worse, there was a right answer. And if you get the wrong answer, lawyers are not going to be happy about it. You know, I had been a lawyer myself but also been a lawyer for a decade. And every time we made the smallest mistake in anything that we did, we heard about it immediately, right? And so, I had that voice in my head maybe as I was going through this process. And that, that how was the learning from the 10 years of slogging through pre-LMS.

you're like, No, it has to be 100%. Oh yeah, oh yeah, that's probably true of way more domains than we realize actually. It could be um, because and the other thing that we were thinking about a lot is you can lose faith in these things really quickly, right? You have one bad experience, especially if it's your first bad, your first experience is bad, and you're like, You know, maybe I'll check on this AI stuff a year from now, especially if you're like a busy lawyer, not a technologist. So we knew you had to make that first encounter, the first week, really, really work for the lawyer, or else they're not going to invest in it deeply. So let's talk a bit about OpenAI 01, because it is a very different model. I mean, up to this point with GPT-4 and all that previous generation, the analogy in terms of the intelligence is sort of the kind of system one thinking and the Daniel Kahneman type of uh intelligence, right? He has this whole economic theory, won the Nobel Prize around it, system one.

thinking is just very fast, is kind of these decisions that humans make very intuitively and based on patterns, and LMs are fantastic at that, but they're terrible at the executive function. Because what I'm hearing with all the stuff that you're describing is kind of you're just giving the LLM like executive function is like how do you think, right? How do I manage you? It's really that slower thinking, and I think AAN is exciting. We haven't seen things built yet because it just got announced a few days ago, right? I think it's getting to that system to thinking, and I think this has been a big area of research, which I saw a lot in news a year ago, where a lot of the researchers were excited to unlock this because this is the missing piece to our AGI. Let's talk about what are your thoughts on one and how this changes. So, first of all, I think it's a very impressive model. Like with other things, we gave it the kinds of tests.

that we knew were failing and the degree of its not just math degree of thoroughness, precision, intelligence applied to some of these questions. And sometimes it's the stuff that you wouldn't expect you need a super smart model to do. Like in one of the tests that we run, we give it lawyers' real legal briefs, but we edited very slightly some of that lawyer's quotations to the case to make it a wrong quotation or wrong kind of summarization of his case. So, this like 40-page legal brief, you alter things with just adding the word like not, can change the meaning of something entirely, right? And then we give the full text of the case as well to the AI, and we say, Well, what did the lawyer get wrong about this case, if anything? And literally every LLM before that would be like, Nothing, it's perfectly right. And it's just not a precise thinker about some of the very nuanced things that we altered.

about the brief to make it slightly wrong, and then one got gets it like immediately like you said, like it thinks actually for a while, like it sits there for a minute, you're like, is this anything thing on, you know, like, but then it starts answering, and it's like, oh, well, you know, changed an and to a neither nor. So, those are the kinds of tests that you kind of expect, even frankly earlier AI like LMS to be able to pass, but just could not. And all of a sudden, o1 is even doing these things that take like precise detail thinking. Obviously, we don't have the internals on o how o1 really works. We have, you know, this broad idea of chain of thoughts seemingly. We know that if OpenAI had a giant corpus of internal monologue of people thinking through doing things step by step, o1 would be even a lot better. It sort of rhymes with the thing you did to, you know, put your first step on the moon, right? Like you, it rhymes with break it down.

into you know uh chunks where you can get to 100% accuracy instead of just throw it all in the context window and you know maybe magically it will work. Yeah, do you think that that's what's happening then? I think there's a good shot that that they've had, you know, maybe change what their contractors are doing instead of just doing, you know, input in answer out. They were doing input in how would I think about solving this problem and then answer out. But then it, you know, the interesting thing is then it's kind of limited by the intelligence of the people writing those instructions. And one of the things that we're investigating for what it's worth with o1 is can we prompt it to tell it what to think about during its thinking process and inject, like again, like we've hired some of the best lawyers in the country. How would the some of the best lawyers in the country think about solving this problem? And maybe, you know, we have no conclusive evidence one way or the other yet that this dramatically

improves things is so early, uh, and just not enough time yet has passed. There's a chance that one of the new prompting techniques with L1 is teaching it not just like how to answer the question, what examples of good answers look like, but how to think. And I think that that's another, like, really interesting opportunity here is, um, injecting domain expertise or, um, just your own intelligence. I'm just so thankful because I think you're sort of sharing the breadcrumbs and, you know, where there are a great many other spaces where this technology is just beginning. I mean, you go to pretty much any company, people have no concept of what's just happened. Yeah, like, they actually literally still repeat all of those sort of tired tropes of, Oh, you better be fine-tuning, or all the... I mean, these things are just not connected to, like, what we're seeing day-to-day with startups and founders trying to create things for users. What I'm kind of glad for is that we get to actually share

this news like this knowledge 'cause like even the things we talked about you know hey you should probably do evals like there's a lot of alpha in getting to 100% not just 70% these are sort of the breadcrumbs that will actually go on to create uh all of the billion dooll companies maybe thousands of them actually we hope so I mean I think that you're about start to see a lot of other fields like law really level up when you don't have to spend you know millions of dollars in six months literally in a basement reading document by document by document right when you when you actually can just get past that and get just the results and now you're thinking strategically and intelligently and the unlock for these companies I mean they currently pay again millions of dollars in salaries for these jobs to be done each of them right so for any company to come out with a AI that can do even % of that the value is like really there and I just want to encourage people to not

kind of give up based on those tropes, right? Like, Oh, it hallucinates too much. It's too inaccurate. It's do whatever there's for an example of anything. It's like there's a path, and you can do it, and there's some good news in that. You know what? The jobs aren't going to go away. They'll just be more interesting. That's what I think. Yeah, well, with that, we're out of time, but Jake, thank you so much for being with us. Thanks for having me. See you guys next time. [Music] Now [Music]

Why Vertical LLM Agents Are The New $1 Billion SaaS Opportunities

Videos to help you build a successful startup. Subscribe for startup advice, founder stories, and a look inside Y Combinator.

What is Y Combinator?
We invest $500,000 in every startup and work intensively with the founders for three months. For the life of their company, founders have access to the most powerful community in the world, essential advice, later-stage funding and programs, recruiting resources, and exclusive deals. 

Visit ycombinator.com to learn more.

Video Team:
Zach Both - Video and Content Lead
Ryan Loughlin - Senior Producer



Y Combinator

사례 텍스트와 AI를 활용한 제이크 헬러의 여정

Case Text의 창업자인 Jake Heller는 0달러에서 10년 만에 기업 가치를 1억 달러로 급성장시킨 후 GPT-4 출시 후 6억 5천만 달러의 투자금을 회수하는 데 성공했습니다.

그는 이러한 성공의 비결로 AI와 자연어 처리에 대한 집중적인 투자를 꼽으며, 이를 통해 회사의 초점을 GPT-4를 기반으로 한 새로운 제품인 공동 변호를 개발하는 데로 전환할 수 있었습니다.

컴퓨터 과학 교육을 받은 변호사라는 헬러의 독특한 배경은 기술을 통해 법률 분야에 혁신을 일으키고자 하는 그의 열정을 이끌어냈습니다.

사례 텍스트의 여정: 어려움에서 성공으로

Case Text는 기술 프로젝트에 시간을 할애하는 데 어려움을 겪던 변호사가 설립한 회사로, 법조계에서 창업으로 방향을 전환하게 되었습니다.

초기에 변호사가 판례 주석을 달기 위해 사용자가 생성한 콘텐츠에 집중했던 이 회사는 실패로 돌아갔고, 더 나은 기술과 사용자 경험을 위해 자연어 처리와 머신 러닝으로 전환하게 되었습니다.

변호사들이 업무의 근본적인 변화를 인식하고 Case Text와 AI 솔루션을 위한 협업을 모색하면서 ChatGPT와 GPT-4의 출시는 전환점이 되었습니다.

GPT-4가 공동 위원회의 성공에 미치는 영향

Co-Council의 제품 시장 적합성을 향한 여정은 GPT-4의 출시로 가속화되었고, AI 법률 비서에 대한 수요 급증으로 이어졌습니다.

서버 문제, 채용의 어려움, 언론의 큰 관심으로 인해 회사의 빠른 성장이 두드러졌습니다.

출시 두 달 만에 로펌의 법률 업무를 간소화하기 위해 AI를 혁신적으로 사용한 덕분에 Co-Council은 6억 5,000만 달러의 투자를 유치하는 데 성공했습니다.

성공적인 제품 만들기: 창업자의 여정

이 회사는 전년 대비 70~80%의 성장률과 1,500~2,000만 달러의 ARR을 기록하며 빠르게 성장했습니다.

창업자는 직접 제품의 첫 번째 버전을 구축하고 초기에는 팀을 소규모로 유지하는 등 실무적인 접근 방식을 취했습니다.

초기 고객 참여와 실시간 피드백은 회의적인 팀원들을 설득하고 제품 개발을 추진하는 데 결정적인 역할을 했습니다.

법률 연구용 AI에 대한 팀의 접근 방식은 이전 모델 버전에서 보다 효과적인 워크플로우로 발전했습니다.

이들은 사용자의 문제를 이해하고 원하는 결과를 얻기 위해 거꾸로 작업하는 데 중점을 두어 프로세스를 개별 프롬프트로 세분화했습니다.

각 단계마다 '좋은' 결과를 정의하기 위해 일련의 테스트를 작성하는 등 팀의 노력은 궁극적으로 더 강렬하고 철저한 테스트 프로세스로 이어졌습니다.

프롬프트에서 테스트 중심 개발의 중요성

테스트 중심 개발은 언어 모델의 예측 불가능한 특성을 관리하는 데 도움이 되므로 AI를 도입하는 데 있어 매우 중요합니다.

고객의 문제를 해결하기 위한 전체 애플리케이션을 구축하려면 독점 데이터 세트와 특정 통합을 비롯한 여러 계층이 필요합니다.

법률 업무와 같은 미션 크리티컬 사용 사례에서 정확성과 신뢰성을 달성하려면 다양한 에지 케이스를 해결하고 명확한 단계별 프롬프트를 제공해야 합니다.

AI에서 개발 프레임워크의 중요성

개발 프레임워크는 AI의 실수 패턴을 파악하고 더 명확한 지침을 통해 성능을 개선하는 데 도움이 됩니다.

연사는 AI의 정확성에 대한 신뢰를 구축하기 위해 초기 테스트를 통과하는 것이 중요하다고 강조합니다.

직관에 의존하는 대신 보다 구조화된 접근 방식을 사용하는 것으로 전환하는 것은 특히 법률과 같이 중요한 분야에서는 매우 중요합니다.

새로운 모델인 OpenAI 01은 보다 철저하고 정확한 의사결정에 필요한 실행 기능을 제공하는 것을 목표로 하며, 잠재적으로 AGI의 부족한 부분을 채워줄 수 있습니다.

이제 AI는 단순한 입출력 모델을 넘어 정밀한 디테일 사고와 문제 해결 능력을 갖추게 되었습니다.

새로운 접근 방식은 업무를 세분화하고 도메인 전문 지식을 투입하여 의사 결정을 개선하는 것입니다.

이 기술은 전략적이고 지능적인 솔루션을 제공함으로써 법률을 포함한 다양한 분야에 혁신을 일으킬 잠재력을 가지고 있으며, 궁극적으로 일자리를 없애는 것이 아니라 더 흥미롭게 만들 수 있습니다.