LangChain 및 OpenAI를 사용한 반구조 검색-증강 세대

LangChain은 반정형 데이터에 대한 효과적인 검색 증강 생성을 지원하여 혼합 텍스트와 표에서 데이터 추출을 향상시킵니다.

### LangChain과 OpenAI로 반구조 검색 증강 세대 구현하기
* 🔍 텍스트 분할은 테이블을 손상시킬 수 있고 테이블 임베딩은 의미적 유사성 검색에 문제를 일으킬 수 있기 때문에 **기존의 RAG는 텍스트와 테이블이 모두 포함된 반정형 데이터에 어려움**을 겪을 수 있습니다.
* 📚 LangChain 쿡북은 Colab에 맞게 조정된 반구조화된 **RAG 노트북**을 제공하여 사용자가 문서에서 텍스트와 표를 혼합하여 작업할 수 있도록 합니다.
* 📄 비정형 라이브러리는 **PDF 문서의 텍스트와 표를 구문 분석**하는 데 사용되며, 멀티 벡터 검색기는 **더 나은 검색을 위해 요약과 함께 원시 표와 텍스트를 저장**합니다.
* 🧠 텍스트 및 표 요약에 사용되는 언어 모델인 OpenAI GPT-4와 함께 **LangChain의 LCL이 체인을 구현**하는 데 사용됩니다.
* 💻 이 노트북은 필요한 라이브러리 설치, **테서랙트와 팝플러를 사용한 PDF 파티셔닝, 검색을 위한 표 및 텍스트 요약 생성** 과정을 안내합니다.
* 📊 다중 벡터 검색기는 표 요약, 텍스트 요약 또는 가상 질문과 같은 작은 덩어리를 포함할 수 있는 문서당 **여러 개의 벡터를 생성**하는 데 도움이 됩니다.
* 💰 문서 인덱싱을 위한 **GPT-4 비용은 약 2달러**, 쿼리 실행을 위한 추가 비용 등 OpenAI 사용과 관련된 비용이 있습니다.
* 🔗 노트북 코드에는 문서 내 텍스트와 표의 컨텍스트에 따라 특정 쿼리에 응답하는 데 사용되는 **LangChain 표현식 언어를 사용해 RAG 파이프라인**을 만드는 작업이 포함되어 있습니다.

In this video, let us look at how we can implement semi-structured retrieval-augmented generation with LangChain and OpenAI. So many documents contain a mixture of content types, including text and tables. So, conventional RAG may not work well with semi-structured data because text splitting can break up tables, corrupting the data, and retrieval embedding tables also poses a challenge for semantic similarity search. So, if you want to extract data from documents that have a mix of tables and text, your conventional retrieval-augmented generation may not work well. Okay, so I was looking for some kind of RAG-based application for semi-structured data, which has both tabular data as well as text, something like this document, which is a paper on Gemini. So, if you go over here, there are certain tables which are present, right? So, there is this table which talks about an overview of the Gemini 1.0 model family. If you go further below, there are some tables with numbers. So, suppose you want to develop a RAG-based application whereby you can pull in these numbers.

from tables. Okay, so Lang chain has a cookbook, right? Where it has various, you know, Python notebooks for performing various tasks, and one of them is this semi-structured RAG. So, what I've done today is that I've taken the semi-structured RAG notebook and I've adapted it to Colab by making some small changes. So, how can you perform RAG on documents with semi-structured data? Okay, so there is this library called unstructured, which will be made use of to parse both text and tables from documents, which are PDFs. Then, there is this multi-vector retriever, which is used to store raw tables, text, along with table summaries, better suited for retrieval. And you have LCL, which is Lang chain expression language, to implement the chains used. So, that is the idea over here. So, you have documents from which tables are extracted, text is extracted from these tables, a table summary is generated using the LLM, which is OpenAI GPT-4 over here. The text chunks are summarized using, again, OpenAI GPT-4. These summaries are then kind of converted into your

embeddings right and using a multi-vector retriever, you can extract either the table summary or text summary based on your query. So, you can retrieve these raw tables and text and then give it to your LLM to get your answer. That is the idea over here. Okay, so let's go to our Collab notebook to get an understanding of each of these components, right? So, as I said, this Collab notebook is just I've just taken this particular cookbook Python notebook, and I've just converted it so that it becomes compatible to run with Collab. That's the whole idea over here. Okay, so first, we need to install L-chain, unstructured, Tika, all these libraries. So, that is what is being done over here. Okay, then we need to install a couple of libraries for PDF partitioning used by unstructured, which is Tesseract for optical character recognition, and there is this popular for PDF rendering and processing. Okay, so for popular, this is how you install it. For Tesseract, this

is how you install it: apt install TCT OCR and these things, and you also interact P um. You install P Tesseract, which is your wrapper for Tesseract OCR. Okay, now why is OCR required over here? Sometimes PDFs will have images, and you cannot actually pass it properly, right? That is why this particular Library, unstructured internally, makes use of TCT, okay, to do OCR on your PDF files, right? Your PDF file could be a mix of text and images, okay? You also require nltk, which is in turn used by the unstructured Library, okay? So, the unstructured library is a library that helps with parsing of different difficult-to-use formats like HTML, PDF, CSV, PNG, pptx. So, this particular framework effectively extracts and transforms complex data for use with every major Vector database and LLM framework. That's the idea, okay? And what is multiv Vector retriever? Multiv Vector retriever helps to create multiple vectors per document, which can include smaller chunks. You can split a

document into smaller chunks and embed those. You can create a summary for each document, embed those, or you can have some kind of hypothetical questions for each document. So, multi-vector retriever, it is, you know, beneficial to store multiple vectors per document. Okay, that is where this multi-vector retriever helps. Um, so there is this notebook which explains how to do it, so you can look into this as well, right? Going back to the code over here, so I have to install NLTK as well, right? Then, uh, you know, uh, the original cookbook notebook was on the Llama 2 paper, but here I'm trying it with the Gemini paper. So, this is a Gemini paper from, uh, Google, right? So, okay, so first I set the path of the content, so basically I uploaded this paper over here. All right, uh, and then what we do is that we actually, uh, over here, uh, with this particular piece of code, we actually are partitioning the PDF. So, this is the place where unstructured, uh, Library tries to find.

image blocks right. It uses a layout model to get bounding boxes for the tables and find titles or subsections of the document. Then it does post-processing. So basically, here is where it tries to actually parse your PDF document, okay? It tries to identify document sections like introduction, then it builds text blocks that maintain sections while also honoring the user-defined chunk sizes, okay? So unstructured users partition PDF, which segments a PDF document by using a layout model. The layout model makes it possible to extract elements such as tables from PDF. That is the idea over here, okay? So once you get the raw PDF elements from your... So the change which I have made over here is that we need to install Tesseract, and we also need to specify the path for p Tesseract over here like this, okay? For the Tesseract command, you need to specify the path, otherwise this fails, okay? So once you do this, we can get the elements which are present over here. You know, in the partition PDF, we can examine them. So if you look at over here when you examine

It says there are composite elements, which is 55, and unstructured, you know, basically tables are 16. Tables are present in this document. Okay, composite elements are aggregated chunks of text. Okay, so again, you can look at it and say it prints out the number of table elements, 66 text elements. So here you are splitting the table elements and text elements. You're extracting separately table elements as well as text elements from your document. Okay, using this unstructured library. Okay, so once you do this, then you can use this multi-vector retriever to produce summaries of tables and optionally text. Okay, with the summary, we will also store the raw table elements. The summaries are used to improve the quality of retrieval. That is the idea over here. The raw tables are passed to the LLM, providing the full table context for the LLM to generate the answer. So the idea is that if you have a table like this present over here, all right, okay, or a table like this, all right, so the idea is that you create a

summary of this table okay by passing this context to an LLM, which is OpenAI GPT-4 over here. You also store this table in its raw form, and when a query comes, you retrieve based on the similarity. You retrieve that this information is present in the table, and you pass this raw table to the LLM so that it can extract the particular response which is required, okay? It can generate the particular response which is required. That is the idea over here, okay? So, you also need to install this LangChain OpenAI, right? And then you need to set your OpenAI API key over here. And then what is done over here is that you are creating a chat prompt template, a string output parser, and from LangChain OpenAI, you are importing chat OpenAI. And here you are creating a prompt. Prompt is your assistant tasked with summarizing tables and text. Give a concise summary of the table or text, table or text chunk. That is the element which you are going to send, right? This is a prompt which chat prompt template.

take this prompt text and it creates its a prompt. The model here is GPT 4. Okay, and then summarized chain. So B basically, you're creating a summarized chain, which is that for every element. This is the prompt. This is the model and get the string output. So this is the summarized chain. Okay, now here what you are doing is that you are applying this summarized chain to the tables. So basically, you are creating table summaries over here. Okay, then you are creating text summaries over here. Now that your text summaries and table summaries are created, these need to be stored in a vector database. Okay, for which you need this chroma DB. And then you will use multivector retriever with summaries to create an in-memory store to store the raw text and tables. And vector store stores the embedded summaries. That is the idea over here. Okay, so here you are creating that vector store as well as, you know, the in-memory store, which is for the parent documents. Right, uh, here you are creating

the retriever. Okay, then you are adding the text. Okay, and the embedded summary of the tables, as well as the text summaries, are also added over here. Okay, so to the retriever, you are adding the text over here. Okay, summary text. You're also adding the summary tables in the vector store. Okay, that is the idea over here right now. You can create a rag pipeline again using this, this, you know, chain over here using the chain expression language of Lang chain. So here the template is answer the question based only on the following context, which can include text and tables. There is a context, there is a question. Okay, so here is the prompt which has been created, and the model is again GP4 over here. And here you are creating your retrieval-augmented pipeline where the context is retriever, question is runnable pass-through. That is how you are creating it over here, and the prompt is this prompt model. And from the output from the model, you are doing the pairing, the string. Okay, so in this way, you have created this rag chain.

over here. Okay, now what you're doing is that you are invoking the chain. So, I asked a particular question like, What is the performance of Gemini Ultra performance on the, you know, it's actually wrong over here, performance, performance, one performance is enough. Okay, What is the performance of Gemini Ultra on the mmu Benchmark per discipline as per table 8? Okay, Why did I put as per table 8? So, when I tried it for the first time, it did not really work. Okay, it was actually present in this table, you know, table 8. So, I wanted these numbers, but it did not work. Then, I explicitly specified it, table 8, and I wanted the performance of Gemini on, you know, this Benchmark based on each category. So, I wanted these numbers to be pulled out. That was the idea, and it actually pulled out saying that the performance of Gemini Ultra on The mmu Benchmark per discipline as per table 8 is as follows, and it pulls these numbers. Okay, quite accurately, you know, it pulls these numbers from the particular table. Okay, then I said

give an overview of the Gemini 1.0 model family. Right, so basically I wanted to extract that from, you know, this table, but the text is also present in other places. So, it actually gave me a summary which may or may not have been pulled from this particular table because this is directly text, right? Um, so basically, what is ultra, what is pro, and Nano? That is what I wanted to pull. So, it says that multimodels by it kind of creates a summary over here and says that the models are evaluated. May not be pulled from the table unless you look back at the trace or something. But then, I again wanted to see if it works on tabl. So, I asked like, what are the results of automatic speech recognition tasks on YouTube? So, here, if you go to, uh, this particular, uh, you know, table over here. I didn't specify the explicit table over here. I just asked this question whereby I wanted to, uh, evaluate audio, this thing, understanding. Um, basically, I wanted to evaluate on this particular thing, right? U. I wanted to extract you.

know this information right. What is the performance of Gemini on this automatic speech recognition on YouTube? Because I couldn't find that in this paper, in other places, exactly these numbers. It was present in this table, and it is kind of summarized. Um, so that was the idea. So, it kind of pulled it, saying that the result of automatic speech recognition task on YouTube as follows: G Pro had a word error rate of 4.9, Nano had a swing, Whisper had... Okay, so it is actually basically pulled that line from the table. Okay, and exactly it is given that answer. Okay, so this is a code for how you can do semi-structured retrieval-augmented generation with Lang chain. Okay, and OpenAI GPT-4 in Lang chain. The concepts used are the multi-vector retriever and the Lang chain expression language. Okay, uh, and making use of the unstructured IO library to pass your PDF documents. Okay, the PDF documents are a mix of structured and unstructured data. Uh, so this is a notebook, uh, using which you can improve your retrieval-augmented.

generation on semi-structured data. Okay, there was a cost involved in using OpenAI. If you look at the cost, you know, for indexing this document, the embedding cost is very less, but the GPT-4 cost over here is close to, you know, around U $2. Okay, for running a couple of queries over here, along with all the summary generation of your structured as well as unstructured tables. Okay, so for running this notebook, it cost me close to $2 over here. So, there is a cost associated with, say, GPT-4 over here. I hope you like this particular video on this, you know, unstructured data or semi-structured data retrieval augmented generation. If you like the video, please like, share, subscribe to the channel. I'll be sharing this notebook along with all the relevant links in the description of the video. Do check out. See you in another video.

Semi-structured RAG with LangChain and OpenAI GPT-4 RAG on tabular data , semi structured documents

Rithesh Sreenivasan

LangChain과 OpenAI로 반구조 검색 증강 세대 구현하기

텍스트와 표가 혼합된 반정형 데이터는 기존의 검색 증강 생성(RAG) 방식에 문제를 야기합니다.

LangChain의 쿡북은 반정형 RAG를 위한 Python 노트북을 제공하며, 이 노트북은 Colab에서 사용하도록 조정되었습니다.

이 프로세스에는 비정형 라이브러리를 사용해 데이터를 구문 분석하고, 저장용 멀티 벡터 검색기를 사용하며, OpenAI GPT-4를 사용해 표와 텍스트 요약을 생성함으로써 궁극적으로 사용자 쿼리를 기반으로 효과적인 데이터 검색을 할 수 있습니다.

PDF 처리를 위한 OCR 설치 및 사용

텍스트와 이미지가 혼합된 PDF에는 OCR이 필요하며, 비정형 라이브러리에서는 이를 위해 Tesseract를 사용합니다.

비정형 라이브러리는 HTML, PDF, CSV와 같은 어려운 형식을 파싱하는 데 도움이 되며, 다중 벡터 검색기는 문서당 여러 개의 벡터를 생성하여 데이터 처리를 개선합니다.

이 라이브러리는 레이아웃 모델을 사용하여 PDF를 분할하고, 표와 같은 요소를 추출하고, 효과적인 구문 분석을 위해 문서 섹션을 식별합니다.

표 및 텍스트 요소 추출 및 요약하기

문서에는 55개의 복합 요소와 16개의 표가 포함되어 있으며, 총 66개의 표 요소와 66개의 텍스트 요소가 있습니다.

비정형 라이브러리는 문서에서 표와 텍스트 요소를 별도로 추출하는 데 사용됩니다.

멀티 벡터 리트리버는 테이블과 텍스트의 요약을 생성하고 원시 테이블 요소를 저장하여 LLM(OpenAI GPT-4)이 응답을 생성할 수 있도록 컨텍스트를 제공합니다.

데이터 저장을 위한 요약 체인 만들기

요약 체인은 입력 데이터에서 표와 텍스트 요약을 생성하여 설정됩니다.

이러한 요약은 원시 텍스트와 표의 인메모리 저장을 위해 설정된 멀티 벡터 리트리버와 함께 Chroma DB를 사용해 벡터 데이터베이스에 저장됩니다.

검색 증강 생성(RAG)을 위한 파이프라인이 생성되어 제공된 컨텍스트에 따라 질문에 답하기 위해 요약을 사용합니다.

Gemini Ultra에서 성능 데이터 추출

사용자가 표 8을 참조하여 분야별 MMU 벤치마크에서 Gemini Ultra의 성능에 대해 문의했습니다.

시스템은 표에서 성능 수치를 성공적으로 가져왔습니다.

0 모델 제품군에 대한 개요를 요청했으며, 표에서 직접 요약 정보를 가져오지는 못했지만 원하는 정보를 제공했습니다.

또한 사용자는 오디오 이해도를 평가하기 위해 YouTube에서 자동 음성 인식 작업의 결과에 대해 질문했습니다.

YouTube의 자동 음성 인식에서 Gemini의 성능

유튜브의 자동 음성 인식 작업에서 제미니의 성능은 단어 오류율 4.

9의 G Pro와 스윙의 나노로 나타났습니다.

이 정보는 논문의 표에서 요약한 것입니다.

Lang 체인과 OpenAI GPT-4를 사용한 반정형 검색 증강 생성용 노트북은 인덱싱과 쿼리 실행에 총 2달러 정도의 비용이 들었습니다.