Imagine you just opened an electronics store, you're hiring some employees. You need to make sure your clients have a good experience as they walk into the store, hopefully purchase more products. And you need to standardize all of it. How do you go about doing that?

As part of this video, we're going to go over the fundamentals that will empower you to make the right decisions when it comes to updating and tweaking your LLMs for your requirements. We will do this from a perspective of context optimization and model optimization.

Context optimization is essentially the window or the text that the model is going to take into account when it generates the text, and model optimization is actually updating the model based on specific requirements. Now let's go back to our store. We have hired our first employee.

A generalist, polite enough, but you won't just let him loose in the store. You want to give some guidelines to this person. So always greet the prospective clients. Make sure you are polite and based on the question they ask, give them the top three options.

Maybe there's some sales going on that may be relevant to the client, etc., etc.. Similarly, in the context of an LLM, they have this thing called prompt engineering. Prompt engineers give very clear guidelines on what they expect from the model.

You can do so by giving some text. You can also give some examples like input and output so that the model can understand what you are really looking for? You can also help the model break down a complex problem into sub-points and make sure it's kind of understanding what you're going after in the long run.

This is called chain-of-thought prompting. Our employee is doing well, but getting inundated by new information coming from all the new devices. That smile can turn into a frown really quickly because it's hard to be up to speed with all the technology changes coming in.

So you have come up with a strategy where you have created this manual, and this panel has all the updates for all the different gadgets coming in. So you're good. But you kind of expect the employee to read that document every time a user asks a question.

So you have devised a strategy based on the question. You retrieve some of the pages from the manual, give it to the employee, and its answer comes back to it. That, in a way, is like RAG, retrieval augmented generation, which allows you to connect this LLM to your data sources to make sure that you're getting the right answers.

This can address things like hallucination as well, because it can really refine the prompt in doing say you need to give the answer only from these specified documents. So it's a really powerful tool as well. Now, going back to our store, business is doing really well.

And we need to hire more employees. That's great. But it was already hard with one employee. How do you make sure you standardize the behavior for all three of them? Being polite can mean different things to different people. Secondly, your customers are getting more savvy.

They're asking more specialized questions, asking how to fix things. So just reading off a guide is not going to do it. What usually is, is you need them to go through a training school, be it from a sales perspective or technical perspective, to really make sure that questions are answered.

That is like fine-tuning. Fine-tuning allows you to actually update the model parameters based on your data to ensure that you influence the behavior of the model and also make it specialize in a specific domain as well. Now, remember in the beginning I mentioned we are doing this in the lens of context optimization.

And LLM optimization. So all that means is that RAG and PE are essentially taking all the information you need the model to know beforehand, passing it over for the model to make its deduction, generate the text and come back to it.

Fine-tuning is actually optimizing the model to ensure that you're getting the right responses with the right kind of behavior that you would need. This addresses the two key problems we keep hearing from practitioners on why they're reluctant to move into production model behavior,

How would you really modulate the model output both from a text perspective as well, from kind of the vernacular and the qualitative aspects, if you will, and then the real-time data access, how quickly can you get the model to answer a question from real-time data as well as ensure that it's accurate and relevant to the user?

So let's summarize that discussion so far at five points. So this one is this whole technique or set of techniques is additive. So they're all working and complementing each other. The first two, RAG and PE, are done in the context of the context window optimization.

Fine-tuning actually updates the model parameters. This is important because the token window is limited. So the more text you add to it, there can be more noise. So you need to be careful about what you're passing to the model. Secondly, on the model, while it may be expensive, the more you spend on the data and actually update the model with good quality data,

you can then use a smaller LLM, instead of a bigger LLM and save costs in the long run as well. The second one is always start with prompt engineering. This is one of the most powerful and agile tools that you have in your repository to ensure that, 

a) you understand whether even having an LLM based solution is right for the kind of data that you have, the end users, Is the baseline model accurate, And all the work that you're doing, even the trial and error can actually be used for fine tuning. 

So it's really, really worth it. The third one is also important. People start worrying about the context window optimization too soon. So focus more on the accuracy versus the optimization. So what I mean by that is as you get closer to the right answer, especially in the context of window optimization, 

keep looking into the right answers and then start seeing different strategies on how you can reduce the window. The fourth one is people say that data quantity is really key for fine tuning. Yes, that's important. But I would take data quality, DQ, over quantity, honestly. 

This is really valuable because you can really start a good example of fine tuning by just 100 examples. Of course, that differs from every use case, but really focus on the quality. Some of the output that we get from prompt engineering is also going to be very important. 

This brings me to my last point. You need to be able to quantify and baseline your success. Just saying that the answer is good enough is not going to cut it, especially when you try these techniques and try the nuances. The permutations between these three can be huge. 

So you need to make sure from an accuracy perspective, precision perspective. Again, going back to the context optimization, if you're using RAG, not only is the answer important, it's also important what kind of documents you got from the vector database.

This will help you reduce latency. So a lot of really good solutions that you can get if you can start really quantifying everything and what success looks like for you. So going to our diagram here, the two key commonalities between all three is going to have increased accuracy.

Reducing hallucinations. So not making up answers. Start with your PE Prompt Engineering will help you really ensure that you have the right solution. So really quick iteration, super valuable RAG is going to help you connect your context window to external data sources.

You can give it some guidance as well, and fine tuning actually changes the model behavior where you can control it more and can become a specialized model in a specific domain that you have. In terms of the commonality between RAG and prompt engineering, context window optimization is key.

Of course it is constrained by that. So as you look for accuracy, you need to focus on how can you optimize that as well. Between prompt engineering and fine tuning wise, they both are kind of inferring some model. They do it in different ways.

So prompting and giving some guidance on responding to three points, fine tuning almost guarantees it in the vernacular that you want it. Finally, between RAG and fine tuning, you can incorporate the data sources, but really think of RAG as the short term memory and fine tuning as a long term memory for what you're trying to do.

So if I were to summarize, context optimization is super valuable. It's one of the easiest and first routes you should take to optimize an LLM model. The second one is, once you've decided, okay, you've optimized it, 

but you're saying you're getting more and more end users, latency is becoming a problem, and now you know, okay, you know what? I can fine-tune my use case a bit more. That's where you use fine-tuning. This will help you really specialize the model.

It won't be a generalist, so there's a risk there. But as you focus your use case more, fine-tuning is the right way to go. So if I were to summarize the discussion, as you know, all three techniques are really powerful.

But if you see it from the lens of context optimization, focusing on all the words and things you want to send to the model before it generates text, it's limited by the number of tokens. So the more you increase that, there's going to be more latency, there's going to be maybe more downtime,

and the more documents you bring in, it could actually create more noise for the model because it really doesn't understand that specific data. However, on the other hand, if you have a model and you know, you have specific vernacular, very specialized domain,

medical, financial, legal, etc., etc., fine-tuning is an option, but you can actually update the parameters of the model using your data. As I mentioned before, you can look at the input and output you got from prompt engineering and make the model a more specialized expert.

It can also help you control the model behavior, which is super important when you talk about corporations and them using LLMs for their end use, those solutions as well. Prompt engineering and RAG. Again, the context window is the best way to kind of start and really ensure that you understand the right way to go.

And you can augment it with fine tuning at the appropriate stage as well.

Context Optimization vs LLM Optimization: Choosing the Right Approach

Whether it’s AI, automation, cybersecurity, data science, DevOps, quantum computing or anything in between, we provide educational content on the biggest topics in tech. Subscribe to build your skillset, learn about new trends, and gain insights from IBM experts. 

This channel follows the IBM Social Computing Guidelines.



Context Optimization vs LLM Optimization: Choosing the Right Approach

Context Optimization vs LLM Optimization: Choosing the Right Approach

Scripts