Hi, my name is Wale Kadus. I'm the head of engineering here at AnyScale, and I'm really excited to talk to you today about the combination of LangChain and Ray. What we'll be talking about today is showing you why LangChain is such an

awesome library and why it's so popular within the large language model community. And then I'll show you a little bit about how Ray complements LangChain by allowing you to build indexes more quickly, serving LangChain on the web directly, and allowing you to put models and the serving in the same

place to keep latency nice and low. So let's imagine now that I'm I'm trying to build a search application, and this is part one of three parts that we're going to have. Part one is this part where I'm just going to build like

a little search engine. In part two, we'll look at enhancing that to be to use large language models to summarize those findings, but really just even doing these basic things with the search engine shows what's possible. So imagine that I'm trying to make docs.ray.io more

accessible. Well, the first thing that I can do is I can use the power of large language models to create a really nice semantic search engine that isn't just finding words but is actually finding meaning. And we're going to talk through how we do this uh in a moment,

but it's really exciting just how easy this is. I remember when you wanted to build a search engine, you used to need lots of machines and lots of organizing, and you still wouldn't get great results. It's just amazing to me how how LangChain makes this easy, so

the process that we're going to go through is essentially that we're going to start with um our docs.ray.io website, just scraped it, we're going to unload the files into memory and then each page we're going to divide into little chunks and it turns

out that this is going to generate about 48 000 sentences. So each page you can imagine breaking it up into chunks. It's not exactly sentences, they might be slightly longer, slightly shorter. Then we're going to use this sentence Transformer that Hugging Face makes

available to project every sentence into a 768 long Vector. This process of taking a sentence and converting it to a long Vector is called embedding, and so we're going to take those uh that row and then we're going to store them in an index, and so once you have the index

that allows us to then find the information. So I'm now going to go to our code and walk you through it. It shouldn't take very long. The first thing we're going to do is just look at how easy LangChain makes this, so we're going to go here and we're

going to specify where the index is going to go. This is all that I need to specify with LangChain to pull all the docs into memory, and then I define a very simple text splitter here that takes all of those pages and cuts them into little

pieces. And so sure enough we load these documents and we create the documents from them, making sure to keep our metadata. So all we're doing here is we're generating a chunks at the chunks using the metadata and then the final stage is for each of these individual

little chunks, we're going to embed them in the space, put them in FAISS, which is a good vector store, and then we're going to save it locally. Now, we used FAISS, but LangChain actually supports a huge variety of different systems. All right, so so this works really great.

The only issue is it takes a few minutes. Now, obviously, here docs.ray.io is not very large; it's about 130 megabytes, but having something this slow at operating scale could be an issue, and this is where the power of Ray and LangChain

together becomes really impressive. So, what I'm going to do here is I'm going to switch over to my Jupyter lab, I'm going to clear the screen, and we're going to start a process here, and while it's running, I'll come back and discuss

what what that process is doing. So, what we're going to do here is we're going to the first part of the pipeline is exactly the same, but then what we're going to do is split those that effort into, um, eight different shards. We're going to do

those embedding processes, which can be quite GPU intensive, using eight GPUs at once, and then finally, we're going to combine them into a single merged database that we're going to push to the FAISS index. And as you can see, it's running very nicely here; it's still loading the

documents. So, it's loaded the documents into memory, and now we've constructed those eight different shards, and now each one of those eight shards is processing the data at once. So, now you can see that you know there are 48,000 documents or so, each one of the shards is is processing

the data and you can see that they are they're loading the embeddings from disk, but it's doing it once on each GPU. And we can start to see that here if we go look at the cluster, so as you can see those GPUs are looking

really hot and they're now chunking away trying to do this data. This is a machine with eight GPUs and so you can see all of those are working really hard and now very quickly you can see it finishes the work and that process is now complete. Some of

the charts are finished. Some take slightly longer, but from taking five or six minutes it's now uh completely solved. So that's an example of how you can use Ray to speed things up. All right, let's go back. So now that we have our index, how can we serve it and

this is really again one of those instances where Ray can really help you to do something without it being much effort. So what we have here is that the webcast comes in and it's usually just a sentence that we'll see. We transform that one sentence

into a vector just like before using the power of embeddings and then we're going to try to find a set of four most relevant sentences or four sentences that help the user understand and return a response. Now let me just have a look at the source code for that, so what we

can do here is just show how easy this is. As you can see here all that we're doing is we add an annotation for serve deployment in the beginning. We load the database and we load our embeddings and then when the clutch comes in we simply do a self.search result here. This

gets the maximum marginal relevance and then just does some nice thing uh I think uh nice processing all we need to do to start the deployment and go through the code that we put together is just have a command that looks like this it names the file that we're going to

use and has the the vector store and it just names that deployment that we've created sure enough you know it's starting up now it might take a few minutes because it's packaging up everything to make it ready for serving and now the Moment of Truth we've

written a very small program here called query.py that simply takes a request at the command line and sends it to the web page that we've just loaded so let's see how that works I'm going to Simply ask the the engine does Ray support Reserve support

patching and sure enough when we hit that it hits a number of web pages that show you exactly where you need the information about racer support for matching thank you very much folks I hope you see now why I'm so excited about the combination of Lang chain and Ray

um if you want to find out more please come to anyscale.com and there's a lot of information there that you can find to do this as well as a blog post that has this information with links to actual code and look forward to talking to you

Open Source LLM Search Engine with LangChain on Ray

Welcome to the Anyscale YouTube channel! Join us as we explore Ray, an open-source framework designed to accelerate the development and deployment of machine learning applications. From beginner tutorials to advanced insights, our videos cater to all skill levels. Stay up-to-date with the latest advancements and hear from experts through interviews, panel discussions, and keynotes.

Anyscale

검색 애플리케이션 구축을 위한 LangChain과 Ray 소개

웨일 카두스는 랭체인과 레이를 소개하며 랭체인의 인기와 속도와 효율성을 향상시키는 레이의 역할을 강조합니다.

Ray는 공동 배치된 모델과 서비스를 통해 더 빠른 인덱스 구축, 웹 서비스, 지연 시간 단축을 가능하게 함으로써 LangChain을 보완합니다.

이 프레젠테이션은 기본 검색 엔진부터 시작하여 세 부분으로 나누어 검색 애플리케이션을 구축하는 데 중점을 둡니다.

목표는 대규모 언어 모델을 기반으로 하는 시맨틱 검색을 사용하여 docs.ray.io의 접근성을 개선하는 것입니다.

LangChain을 사용하여 시맨틱 검색 엔진 구축하기

이 프로세스에는 docs.ray.io를 스크랩하고, 페이지를 청크(약 48,000개의 문장)로 나누고, 문장 변환기를 사용하여 임베딩하는 작업이 포함됩니다.

임베딩은 효율적인 정보 검색을 위해 인덱스(FAISS)에 저장됩니다.

LangChain은 문서 로드, 텍스트 분할, 임베딩 생성 프로세스를 간소화합니다.

이 프로세스는 청크를 생성하고, 메타데이터를 유지하고, 청크를 임베드하고, FAISS에 저장합니다.

LangChain은 FAISS 외에도 다양한 시스템을 지원하지만, 대규모 데이터 세트의 경우 프로세스가 느릴 수 있습니다.

레이는 랭체인 기반 검색 엔진의 속도 한계를 해결하기 위해 도입되었습니다.

이 프로세스는 8개의 샤드와 8개의 GPU를 사용하여 병렬화됩니다.

임베딩 프로세스가 GPU에 분산되어 처리 시간이 크게 단축됩니다.

병렬화를 통해 48,000개의 문서를 더 빠르게 처리할 수 있습니다.

레이를 사용하면 단일 스레드 방식의 랭체인 접근 방식에 비해 훨씬 빠르게 프로세스가 완료됩니다.

Ray는 웹 인터페이스를 통해 인덱스 제공을 간소화합니다.

이 과정에는 들어오는 문장을 벡터로 변환하고 관련 문장을 찾는 작업이 포함됩니다.

간단한 코드 추가만으로 레이서브를 사용하여 배포할 수 있습니다.

간단한 명령으로 배포 프로세스가 시작됩니다.

데모에서는 배포된 웹 인터페이스로 쿼리를 전송하고 관련 결과를 수신하는 모습을 보여줍니다.

웨일은 랭체인과 레이의 결합에 대한 열정을 표현합니다.

시청자는 코드 링크가 포함된 블로그 게시물과 자세한 정보를 보려면 anyscale.com으로 이동합니다.