great, yeah. Thank you for coming around this time. I'm sure people feel sleepy around this time, so, uh, hopefully this is somewhat entertaining. So, I'm going to tell you, um, you know how we are trying to think about the future of search, which is probably the most, you know, intuitive and important product that, you know, every consumer can use, and how we are approaching it at, uh, perplexity. So, uh, think of this kind of linear story where we just saw like Google work like this, where, you know, I'm searching for like good headphones, and there's just a ton of ads, sponsored links, SEO content, and super tiring, and then you just decided, okay, it'll be nice, what, what it would be, what if it'd be really cool if we can just directly just see the answer and compare different things, and so we just, uh, decided, okay, let's build the world's best, uh, research assistant that can just answer any question, and then we ended up building this, and that's the end of the story. Now, that'd be very disappointing, right?

like you'll be like, Oh, why did I even come for this talk? But so the answer is obviously it. I wish it were this linear. I would think about myself like a genius or something, but uh, it was hardly this linear. So this is actually what really happened. Like, we incorporated one year ago. Um, we start working on text to SQL, which had nothing to do with consumer search. In fact, like, if you went to uh, VCS and asked for money to work on search, they just wouldn't even fund you, right? Like, no matter like, uh, how ambitious you or like how like, how you, it's very hard for you to like convince anybody to fund you. So we, we worked on Texas sequel. Uh, we got lucky enough that really good people like Elon Gill, Nat Friedman, Jeff Dean, all like invested in US, and we were like really just interested in like looking into Enterprise search with llms plus SQL. And, uh, we just couldn't know how to, you know, even do all this stuff company building Enterprise, none of these things like.

nothing was intuitive. There was no expert to answer anything. We knew, so we just thought it'd be great to build stuff that can answer our own questions. You know, that we can like figure out things, basically. I'll just kind of show it all. This looks very like retrofitted, but it's not. I'm going to actually give you screenshots and show you everything. And then we, um, in November, we launched the web search, you know, for our own friends. We had Discord Bots. Then, obviously, end of November, Chad GPT launches, considered the iPhone moment for AI. A week after that, we launched perplexity. And so that's, so this is exactly like, you know, we incorporated on August 3rd, uh, and then like on September 27th, we had this Slack bot. So this is for ourselves. Like we look at the questions we asked, like we're asking like how to start Uber server and Ruby, or like what is SOQL, it's like Salesforce query language, because we were working on like stuff like, oh, how to like search over.

your Salesforce data. How to search or HubSpot data, and we had no idea about any of these things. So, for example, this is me trying to write HubSpot SQL queries as templates for, like, you know, teaching our contractors how to do that because only if I know how to do that, I can teach contractors, right? So then, I don't know how to do that because none of these tables make any sense to me. Uh, they're also complicated. There's just too many things, and people customize it a lot. So there is no HubSpot expert you could go and talk to. So it'd be great if there's a search engine that just answers my own questions. And obviously, Google doesn't answer any of these questions. So that's the difference between what AI can do and what Google can do. So Google can give you links. Google can answer you, like, navigate you to a web page, give you these, like, very trivial factual answers for, like, very simple things, but it cannot answer complex queries, like, you know, what is a life cycle stage mean in a certain table or

like if I were to change this to this thing, like how do I still get information out, like what, you know, all sorts of things that you would have to ask another fellow employee or a friend or a colleague working somewhere else like that. These kind of things are rare, like, you know, AI has really changed the story, so it's creating a new product, new market segment, and not like taking market share from something that already exists. So we found this to be super useful for ourselves, you know, like, uh, Paul Graham is very famous for saying this, and Y Combinator is very famous for this, that you first have to build something that you and friends around you, people around you use, and if you don't even have that, it's very hard for you to find product-market fit with somebody else. And even if you do have something that someone else wants but you never use it yourself, uh, it's unlikely that you'll do a good job at it because you don't really care, like, why, why is that person's problem your problem, right, like, that's it.

it's your problem because you want to make money by solving their problems. But that won't be a long-lasting motivation. So I think it's very important that, like you know, you pass this filter for every startup. That you yourself and your own people in your own company start using it and you learn so much from it. So that this is another example. I, you know, I had no company building experience before starting this and at some point, I realized, like you know, when I would try to recruit people, they would ask me, Do you give health insurance? And like, we didn't even take health insurance by anybody. Like there's no policy. We didn't even have Gusto or Justworks, none of these. So then when you go and pick like stuff, I had no idea what any of these terms meant. There are all these stuff like deductibles, you know, they love confusing you, right? So, so again, Google is pretty bad at these things. If you ask these questions here, you just get like SEO links and like all these insurance providers just pay

Google so much money to get their websites up. It doesn't actually, uh, solve your problems. So I used, I started using it a lot and then, um, what we, what we ended up realizing is that when we ask these sort of questions, uh, I wouldn't be able to trust the answer. So when I would go and ask somebody else who had done this before, they would tell me, no, no, what you said is wrong. And obviously, you know, in hindsight, it's obvious, but you need to get plug into real data, uh, or else you can't trust the answer of, uh, chatbot that's just the language model. So you need to plug it into a search index and, and then my co-founder Dennis is our CTO, like he's just like casually says, I added some Bing search and summarization, and then, um, we were able to ask like real live questions, like what happened CEO, like at that time it was a time when Elon was, uh, taking over Twitter, uh, so he's like, has Elon fired Twitter Executives, and that was when he

fired the previous CEO Parag. So to answer all these questions, gave all these links. It was all like a nice Slack bot. We were all like just back into an AI people, so we didn't have time to build front-end. So we just tested with Slack bots. And then um, it started becoming fun as we made it conversational multiplayer, uh, you know other people could come and ask questions on top of what someone else asked like. So here Nick, our first engineer, he's like, Why did Elon buy Twitter? And then like Dennis goes and asks, Is he taking it private? Uh, so you know, it's all. And I'm very, very useful. Like you can just learn a lot from seeing what other people are asking too. And um, and then our investor Nat Friedman tells us that, you know, like Midjourney is a Discord bar. Like, you know, it shouldn't work on Slack. Like Discord is an expert thing. So you make a Discord bot and the UI is also much better than Slack. So we made a Discord bar and like we started a

small Discord server um and like uh you know it was pretty pretty funny entertaining whoever joined the server and tried out the bot. They would they would be like, Oh, this is so much better than Google. Like, I would use it every day. And we we didn't initially believe that. We thought, Okay, they're just saying it to be nice to us and be encouraging. But uh, they did actually keep using it. So, that gave us a lot of um, you know, hope that this is actually something real. So, note that this was still a time when we were working on um, our SQL Enterprise stuff. So, this was more like a distraction. So, I wasn't sure what to do. Like, I was, Why are we even working on this? There was always that question. But uh, one thing that's true is when something starts working, uh, that there is like a dopamine from constantly improving it. When you see people asking for stuff and it's, you keep shipping, it keeps getting better, you feel the momentum. So uh, my co-founder again, Dennis, added these uh, focuses where you

can search specifically about Wikipedia. And then you can search specifically on back overflow, and it would answer all our coding queries like so much so much better. And so there was something real here, and this was exactly one day before um Chantry launched. And uh that was like, you know, still not sure what we should be doing because we were working on our company was supposed to be doing something else. And we saw the charge GPT launch, you know, it's historic right now you think about it. Um, and then this they broke several myths that you needed to be in Discord, like people don't want chat Bots, like all sorts of things. They basically broke all the existing wisdom at the time. So we were very encouraged by this, and we thought we could still add orthogonal value to what they launched because we had this orchestration of combining live knowledge and the chatbot together. So this is again like Dennis asking Nick if you if he had like a moment to help him with the CSS because we don't know CSS. And then, uh, finally I tried to generate

the code for it with uh chat GPT, and then um, and then that broke obviously. So then you know somehow Nick came and helped us, and then we sent it to one of our friendly investor friends, Daniel Gross. And if you look at this first comment, you need a submit button you didn't even see, because that was a joke because um, you know the latency was so bad, it was like it would take seven seconds or eight seconds to get the answer. And now if you use our product, it's like almost instantaneous. So that's you know how many iterations it's come through. And um, he gave us a lot of good feedback, you know, this is always very important, get your first user to like your product. And then uh, we had this whole thing ready, uh, we were like going to launch it, and then we got The Bravery to launch it. Uh, not Friedman was like go get 100 people, you know, like 100 people to use your product. And so we launched it, and it went really well.

better than expectations. Um, and this was sort of the beginning when we decided, Okay, fine, we had this whole SQL search for Enterprises. Um, we prototyped it on some large relational databases, so we thought like we might as well release something like a public database-powered SQL search and test which one actually works. Is it the web search or the SQL search? So we launched the search over Twitter, like, which actually got, you know, like Jack Dorsey himself to come in, uh, uh, code tweet us saying, This is great, and that got a lot more eyes on perplexity, and people started search using both products. We tracked the usage, and we saw that the regular web search product got a lot more usage. We were still confused whether we should kill this the research or not, but Elon made it easy for us by, you know, changing the API prices to like something so high that nobody can pay anymore except Google. So we decided to just focus on web search, and we made it better. We can't, we made it fully end-to-end conversational.

we suggested follow-up questions, and then, like, you know, I think one thing that, you know, NAD was gracious enough to say is that we ship really fast. That's basically our identity, almost some people give us a lot more credit than we deserve, but I think, uh, you know, that's the only way in which we survived this whole competition with Microsoft and Google trying to do all these same things. So we kept iterating, improving from January onwards, tried many, many experiments, and then the next big update was launching this thing called Copilot, ah, which we consider as like a browsing companion, right? So the way it works is, um, until now, you just ask a question, and the AI, uh, plugs into a search index, gets the answers, and gives it to you in like a direct chatbot UI. We thought it would be cool if it had it's more agentic and comes back to you as some clarifying questions and then, like, gives you the answer. And the clarifying questions it generates is very question-dependent, it's very dynamic, and it's the kind of

questions. It asks can be like multiple choice or single choice. So we innovated on this new thing called generative UI. Like the based on your query, a user interface is generated according to it. And Versel is doing great work now on like, you know, trying to generate websites based on what you want. So we had this idea way before, and like sort of inspired by all the noise around Auto GPT and baby AGI at the time. But our thesis was that like autonomy is not like there yet. We still want like these agents to work along with us. So we designed a product that way. And this sort of basically became our, you know, USP of our Pro Plan. And people use the Pro Plan for like the co-pilot, basically for like more, uh, research on steroids. That's the best way to think about it. And then we made it even faster recently with OpenAI's GPT-3.5 fine-tuning API. So on the left, you see the fine-tune GPT-3.5 model. That's

running copilot on the right, you'll see GPT-4. So GPD-4 asks you more questions, but not all of them are needed. GPD 3.5 is much faster and just directly gets to the core point. So ideally, we want something that would have the brain of GPT-4 and asking clarifying questions, but the speed and the reliability of GPD 3.5 in terms of user experience, right? So that's what we tried with this fine-tuned model. And in terms of metrics, if you just look at our FT, GPD 3.5 is like the fine-tuned model, and GPD-4s and 3.5 are the original models. Eighty percent of the time, the user is unable to say which model is which, and eight percent, they think the fine-tuned model is better, and like 11, they think GPD-4 is still better. So there's still some room for improvement, but this is going to go more towards the user being

unable to say. And in terms of GPT for us is, um, like fine-tune versus just the original GP 3.5. You can clearly see that it's winning a lot more compared to, like, you know, without fine-tuning. And there's a reason we do that. This is because, like, the, you know, the, the price you pay for output tokens. This is, you know, not even comparable, right? Like, for example, if you look at the input tokens. The prompt tokens, you're paying 3x lower for GB 3.5 compared to four. And for the output tokens, the generator tokens, um, you're paying 6X lower. So, this, the volume of search requests we serve every day makes a ton of difference to us. And it also, in terms of speed, you can see the throughput here and the latency here, um, you know, it's like day and night difference, right? So, all this, like, makes for a great better product experience, like when you're using on your phone with crappy Wi-Fi or you're on your mobile internet. These things matter a lot.

so we thought through a lot of this took the opportunity of an open AI had the fine-tuning API available, and you know shipped this much faster than other people, and, uh, yeah, this got the appreciation of the man himself. He came, said great use in metrics open AI fine-tuning API and, um, yeah. So we also, you know, support other research use cases in the product like we let people upload files and Cloud tools were very much appreciated and known for its long context, and you know, file upload feature. So we didn't just do file upload because so it'll be cool to just add another feature. Our goal is to be the best research assistant anybody can have, and uploading your files and asking questions about it is like one very essential component of any research workflow, right? So we just did this, and we thought claw 2 has much better capabilities here compared to gpd4, so we worked together with anthropic and made this happen, and this also got a lot of positive attention for us. So recently we launched this thing called collections where this is our

transition from you know moving from just a tool to a work end-to-end platform where you can like instead of just pasting our URL somewhere and like being scattered around your slack or Whatsapp or like your you know email somewhere. We just started to be nice if it's all like persistent and like store saved some air and you can collaborate with people. You can also have your own privacy levels in terms of what you want to share publicly and keep it private. So if you want to have like one set of collections for your like you know planning your holiday trip with your friends or family, then that's different from having another set of collections where you're just doing research on companies or you're having like coding queries about some new uh framework that you're learning. All of these can be like you know decoupled right. So this is sort of becoming more like a platform than just the two and that's very important because we see this as a way to like truly differentiate ourselves and have people

coming back to using the product every day. So, this is uh, you know, something that we'll be doing more of in the coming months. And this thing, I want to like, you know, I've said this before, uh, there's a lot of like, you know, different opinions on like, you know, whether being a rapper versus training your own model, uh, what is it, what is like, you know, respected more, like what's needed for a startup. And there's this code, uh, from this Steve Jobs movie where it was like basically asked him, what do you actually do, like, you know, you don't own the, you don't do the hardcore engineering, and he says, I play the orchestra and orchestration, like orchestrating different components, search index, LLM, conversational rendering, the answer, multimodal. It's not easy, it takes a lot of work, even in being a rapper, like, if you do decide to be a rapper, you might as well be a good rapper, right? Like, and then devil is in the details and, um, so for example, like, I was telling some about like The Bard.

extension story on Twitter. And I tried like, Hey, I'm just gonna see if this works. Maybe I'm wrong about orchestration. And I just asked how many flights they've taken in the last two months. And it doesn't even plug into my Gmail to give me the answer. And then there's a user here coming and replying to me, saying that when it worked when working with documents, like, perplexing gives way better answers with our hallucinations. And then similarly, this is another product called uchat. And then I don't even like try to game it. I'll only pick the query they suggested, Write a song about whales in the ocean, and then it gives me a weather widget. Like, I didn't ask about the weather, right? So, it's very important. It's very easy to like say, Oh yeah, we connected to all these plugins, we connected to all these APIs, data providers, and then when you try it, like, it does something unreliable at inference time, right? So that's why like we are being more careful here and trying to do it.

with thought like for example Stanford evaluated um all these generative search engines, and like you have two axes here. One is perceived utility, and the other is the citation F1 score. F1 score is just a weighted average of precision and recall, and you want to be good on both. That is, you want to have a high citation score so that you're backing up what you say in the answer with the relevant links, and you also want to be having high utility scores so that your answer is readable to the user and makes for a good experience. And so you want to be as far as possible in the top right corner, and not too much on like, you know, only on y-axis or x-axis. And you look at it like we were the best prior out of all, all four, uh, at the time of, you know, publication of this paper, and this was at a time when we were only using GPD 3.5, and Bing was using GPD4 as default. So, uh, if we rerun the evaluations with like the

current model, which is much better. I'm sure we'll score even better on these metrics. So, people are seeing that, and you know, these days, people are tweeting about it. At the time we raised funding, a lot of like, you know, it was exactly the time Bing was launched, and we ourselves had a lot of fear, leave alone people around us scaring us that, Oh, like, why are you working on something Microsoft's gonna win here? But I think what we learned over these months is that it's not just about one launch. You got to keep on iterating and keep on improving, and users will eventually realize the better product. So, even though it's not like, Oh, like, you don't need to think like this guy is thinking he's already ahead, but you know, like, we fully understand we kind of have to keep improving the product. But the most important part is establishing the platform, the flywheel, the infra, and like the iteration components. So, that's what we are very happy about doing so far.

and this is like a good summary of like what you know how to best interpret or do. It's basically like Wikipedia and chat GPT had a baby, but the data is not just Wikipedia. It's from the entire internet, so it's a perfect tool for doing a deep dive on any topic. It's built for people like you. You know, we're all interested in diving deep and digging into rabbit holes. So that's the idea. It should excite people like you to use these products. Additionally, we also took this feedback that we don't want to just be a wrapper. So we are trying to serve our own models too. With that in mind, we first launched llama models as part of Perplexity Labs, which is the fastest inference llama right now among any other products like Hugging Face Replicate. So many other people are also serving llama models, but in terms of metrics, we publish our metrics and like

you can actually see the tokens per second time to First token everything right so uh we have the best latency and uh I think this person also evaluated it with so many other people and realize like we are functioning the best so actually I want to do a demo of a model that we just launched today uh in the long perplexity Labs. It's called llama 213b sft so this is a step towards us training our own models too not just serving matters models. So if you remember llama got a lot of criticism for like you know having [Music] like basically being super annoying to use it would just not uh obey our instructions because it's it's just too safe right and then we decided okay that's a good fine-tuning step towards you know if you go back to the original llama and ask it to do the same thing it's not going to answer you all these things so if you ask it like how to kill Linux process he's just going to tell you I'm programmed to be the you know safety and

well-being of people. I can't tell you how to do it all these kind of things or like it's not going to give you uh jokes that you want like all right jokes about any person and we decided that we'd at least try to like fine-tune these models to be more useful rather than going too much in the direction of homelessness. And so this is an experiment we are trying out and we expect to update people more on like our own models and start serving our own models doing the product. So right now already in perplexity, uh, you're getting this, you know, if you're on the Pro Plan, you can pick your models of choice like we have our own models, the cpt4, that's claw two and uh, I think over time, uh, the model that we serve will be the best suited for this particular product. So and we also expect like, you know, not people not to be like people probably won't care enough to just pick the models of their choice but just, you know, how one model that just works and that's our goal that's

why we're trying all these experiments to train and serve our own models um yeah so that's basically it um I think I'm have some time for questions [Applause] so where are you using array here sorry where is the usage of three are you using array any any part of Ray oh raining or yeah so we we consider using it uh for fine-tuning models and we are currently testing our own infrastructure based on Megatron and any scales fine-tuning so we'll have something to say there soon production so nothing in production nothing we can announce probably clear okay okay thank you and also the citations right you pointed the paper I'm user of both of you you.com and perplexity right so they're also improving I see both are equal now so what's your roadmap of coping up with that competition now both are equal because on the same data set which Stanford because I'm a Stanford student I ran the same and both are equal now so what's your roadmap ahead I mean you can share

your results with us. I don't think both are equal, but if it's true, then well, I can comment on it more. Sure, sure, thank you. Um, he um been using perplexity for quite some time. Thanks for building it. Um, I've often noticed that when I ask a question, the citations that come up on perplexity are much higher quality than the ones I get on Google. Google's still ranking the SEO articles a lot more. Yeah, given that you're still searching the internet, finding those data points out, how are you maintaining the authenticity of the links that, yeah, you used to, yeah. So we, we do the, we have our own like page ranks of the web, and we also have, uh, the algorithms are so good at like picking the right, you know, relevant links. So the final component of llms, picking which papers, sorry, which links to site is, uh, is actually like a big gain in relevance ranking. So that's also very useful. Thank you. Um, I have a question. Uh, what are the trade-offs you had to make to get llama?

running so fast because it seems pretty fast. Yes, yeah. So, we--I mean, we basically have our own custom inference stack that's, you know, we obviously look at everything that's helping others like flash attention or using facet Transformer page detention. All these, all these different ideas, um, but we decide to use our own stack rather than relying on hugging face or like wheel of them because that way we can control more. And yeah, so I would say just at some point you just have to go and write stuff at level of Cuda, right, or else it's pretty hard to like make things much faster. [Music] I think you spoke a little bit about the model that you are customizing your own models. How are you in a larger space context? How are you seeing the role of Open Source? Do you imagine, yeah, you'd be going down that route more and more away from chat GPT or you'll still be using cloud or charging, yeah. So I think there's competition with from between, you know, GPT 3.5 fine tuning and, uh, open

source models right. Like when we thought, Okay, Llama is great, and we would rather just went to Lama. Then OpenAI also announced GPT-3.5 fine-tuning. Ah, and so if our goal is just to make the product better, uh, we would do the thing that's cheaper and easier to do, right, in the short term. Um, that said, I still think there are some things you just cannot do without having access to the weights. Um, and maybe some of this could just be the style of the response, um, and like other if you want to make the LLMs like more customizable, more agent tech, all these sort of things, like I think it helps to have your own model. Plus, you don't control the pricing, right? That's the most important part, like, leaving aside like the technical, I mean just think about it from the business perspective, like if you don't control the pricing of something, uh, it's always in a tricky position to be, uh, very, very, and, and it's unclear if like pricing of all these APIs will keep

remaining the way it is today, or it'll go down or go up. So it's good for you to have your own stuff. I think it'll definitely catch up to GPD4 by like mid next year. But by the time maybe they launched GPD5, so there'll be some delay, I think. Sorry we couldn't get around to all the questions, but uh, let's give Arvin another round of applause. [Applause]

Perplexity AI: How We Built the World's Best LLM-Powered Search Engine in 6 Months, w/ Less Than $4M

Welcome to the Anyscale YouTube channel! Join us as we explore Ray, an open-source framework designed to accelerate the development and deployment of machine learning applications. From beginner tutorials to advanced insights, our videos cater to all skill levels. Stay up-to-date with the latest advancements and hear from experts through interviews, panel discussions, and keynotes.

Perplexity AI: How We Built the World's Best LLM-Powered Search Engine in 6 Months, w/ Less Than $4M

Scripts