[Music] Hello, we'll give him a few minutes to hop in and join, but we have a really exciting new uh, really exciting new series, the Agents in the Wild. And today, we have um, Booking.com actually talking about one of their um, amazing agent applications.

Um, so to folks who are joining, we'll we'll kick off, we'll do some intros, we'll go over a little bit of the agenda, and then I'm going to hand it to the Booking team to talk about um, a lot of things about their agent, how they're

building it, how they're evaluating it, kind of Lessons Learned From The Trenches. Um, so uh, for folks who have questions, please feel free to just drop them in the chat, and what we'll do is um, I'll I'll try to find a way to get them

in as as we're going through the presentation today. Awesome. Okay, well, let's kick it off. Let me share my [Music] screen. Okay, let go ahead and oops, awesome. Okay, and go ahead. Okay, awesome. So, let's kick this off. So, today, we have an uh, really

awesome session planned uh, Agents in the Wild, this time looking at one of the agents that the Booking.com team is built um, and with that, let me actually uh, ask some of the Booking team to introduce themselves. Um, Connor, do you want to go

first? Sure. Uh, hi, my name is Connor Ross. I'm a machine learning manager at Booking. I joined Booking about two and a half years ago, and I manage the recommendation platform and the Gen orchestrator, which we'll be talking about soon. Awesome. Uh, Maran, do you want to go

Next year, hi I'm I'm the machine learning manager at Booking.com, leading the content intelligence team which is responsible for different content areas like moderation, J evaluation, gen, and also um supply enrichment using Booking images and unstructured data. Um, yeah, and I'm so happy to be here and

uh, tell you about our gen stuff. Awesome. I'll just go really quick. My name is Aara, one of the founders here at Rise, and I work really closely with the Booking team. Um, uh, my goal is just to, I like kind of the awesome we've been hearing a

lot about the amazing work their team's been doing and cool lessons from the trenches, so goals to kind of hear, hear from the experts here. Uh, with that, let me share a little bit of the agenda. Uh, the Booking team is going to walk

through their use case of why they decided to build what they did, uh, walk through the architecture, design decisions, and trade-offs that they had to make. Um, um, and then talk about that continuous iteration loop. Um, and they'll, they'll kind of share some of the key

takeaways or lessons learned along the way. Um, again, drop questions in the chat, we'll pull them up and and run through it. And with that, maybe Ran, you could go ahead and share the, um, kick us, kick this off. Sure. Okay, so we will share with you uh

the story about the I-tree planner, which is the first gen product uh, we built. Um, and um, yeah, we just did the introduction. I also joined Booking two and a half years ago, and I in general I'm passionate about the travel domain and genl community. Um, so let's see a short demo.

of what is the I3 planner? Um so yeah, so the idea of the I3 planner is that you can start the conversation from the index page. You can, you can press the button and then you have this UI where you can start chatting with our chatbot. Um you can ask about

Tes see, uh different, uh get recommendations for different destinations and then to customize your, uh, uh what you ask for, so you get even a better destination recommendation. Uh we have itineraries, uh recommendation, so we if we start user ask for itinerary, we split

the, uh, the answer by each day and and we also have personalized recommendations, so we ask follow-up questions to better understand the user needs. Uh who you're traveling with, with when you're traveling, and then we give even better, um, personalized recommendations. Um, so given this is the

product where we stop it. Um so yeah, we have an end-to-end flow starting from the indexer page where you can press the banner and try and start a conversation. Given getting destination recommendation, property recommendation, and then once you are clicking the property, you're, uh, you will

be in the property page and you can actually book the property, so from exploration phase to actually booking and and, uh, having, um, conversion. Um so the first question is, is it just another GPT application? Are we doing just GPT calls? Um so the answer is no, like, uh, we

optimize the flow a lot for the travel domain and we build dedicated architecture that we will talk about. Um, we have personalized recommendation system that is more accurate, less hallucinated using internal booking data system to do that, and the internal booking data. Uh we have explainable

recommendation, we explain our recommendation, we enrich them with textual, uh, explanation of why this is the best fit for the user. Uh, we have a reix with Carousel booking funnel, and we build the flow and, uh, and the architecture that it's very easy to

integrate it with Inous model. So if we want to replace a third-party model, a GPT call with, um, Inous model, we can do that. And we also will talk about it a bit, um, so the main components of the I3 planner are these three. We have the gener

orchestration, which Hann will elaborate on, but the main idea here is to, uh, coordinate, coordinate the flow, to manage the flow and have the communication with booking internal services. We have the NLP service, which is managing the LLM task, um, so it's actually doing, we have

different LLM tasks, and we, uh, uh, the service is responsible, responsible for each one of them, um, and we have the recommendation platform, which is responsible, we have our internal recommendation models, um, and the idea here is that we can use our internal system to give

recommendations. Um, yeah, so let's start with the NLP service. Um, so this is the NLP service ecosystem. So from, uh, one side we have the interaction with the Gen orchestration, where we have different tasks, and then the orchestration knows how

to build the flow and how to use each task in a, in a, uh, in the correct way. Um, and the main task we have for the I3 planner, our first moderation, so we have, uh, given a user input, we want to be able to moderate it to see if it's in

appropriate language if it's a PII data. Um, then once we move from moderation, we want to have intent classification to identify what is the current intent of the user. And this is mainly the main hub of the flow, uh, that directs the user to the next steps. We

have judged and asked, uh, LLMs to define if we have enough information to give a user, um, a recommendation. If not, we ask a follow-up question. Um, then we have a way to, given the conversation, to move to transform the conversation from unstructured data to structured format.

Uh, having a JSON with features that then we can feed these features into a recommendation model, internal recommendation model. And we have, for example, Caron enrichment, which is responsible to enrich the recommendation with text with text. Um, this is just an example of the LLMs that we have. We have

more also question answering mechanisms and things like that. Um, and, uh, each block here, uh, is also interacting, uh, with different models. It can be third-party LLMs, it can be in-house LLMs. Um, so we built it in a way that it's very easy to

configure each block, what is the model behind it. Um, yeah, so, um, just to explain about the flow about the process that we did here is first when we built this product, we start simple, like we have these different blocks that I just showed you, and we, uh,

build some very basic prompts for each one of them. And we use prompts to achieve each task. Um, then we, uh, the next step was to optimize these prompts, like to give few-shot examples to use many prompt engineering techniques, chain-of-thought reasoning, to add a lot of

test to these prompts and then we are more confident about them and then we thought about switching from prompt to in-house fine-tuned model uh Andy our own in-house model and why to do that so we have three main reasons first is about accuracy so um once we launch this chat

this chat we get a lot of data from real users and we have real interactions of users with the chatbot and we can annotate this specific chat and task uh we have a our annotators of having a classification of the intent the right intent and then can use this data to

find you a better LLM that fits to booking uh data and by doing that we gain higher accuracy um we also uh we can gain a a latency decrease so having a third party model you are uh um sometimes they they have outages and they're not fully reliable

and by having our own model which we serve on our own resources we can control it much more more um and we actually uh was able to to decrease the latency and of course it's about the price we don't need to pay third party model and uh yeah um so a bit about

training LLMs and what we did here um so there are few options of how to train LLMs you can train a pre-trained model from scratch to take massive data sources a lot of data billions of textual data and train next token

prediction um and then you train your own uh language model to uh on the next token prediction task um there is a option to do supervised fine-tuning and with instruction following so you take a base model that was already pre-trained and you just fine-tune it to your

specific task with an instruction, um, and the third, uh, part is about alignment. How we make sure what we build is also aligned with human preferences. And about alignment, it's like we, um, we use humans in the loop and we give, for example, in a summarization task

we give the annotators, the humans, two summarizations. Now you need to choose which one is better, um, and given a prompt and given a task, we have the chosen one and the rejected one. And then we can use this data in the training process like in RLHF, direct preference

optimization, uh, to train the model to also optimize, uh, these kinds of preferences. Um, so mainly in Booking, we are doing these two steps, um, we are doing a lot of fine-tuning, but also some alignment for, for example, for summarization tasks. Um, so if we deep dive

a bit more, when we do fine-tuning, we use parameter-efficient fine-tuning, which is, uh, a Hugging Face library where you can do not full fine-tuning, but just to have an efficient, uh, fine-tuning by not, by just running part of the model

weights and not all of them. So what we're doing here, we are taking already a an open-source model for the pre-trained phase and then we want to fine-tune, um, and given an instruction on a specific task. So we basically have tried two options. One is a

prompt tuning, uh, the idea of prompt tuning is to inject virtual tokens to the prompt, and then you just train and fine-tune these, uh, these tokens. All the rest is frozen, and then you make your, um, model to be more, more customized, more relevant

to the to your prompt um the second one is Laura which the idea here is to add adapter layers which are low rank matrices um and then you train just these layers and by doing that by the way even with C which is quantized adapter uh we train less than 1% of the

model parameters instead of doing full fine tuning and this process is much more efficient much much cheaper uh and uh much uh easier to maintain and by doing that we did it for example for intent detection so now we don't use GPT for intent detection which is the Hub of the

process we you we use um in-house LLMs and by doing that we boost the accuracy uh by 133% and also we gain five times latency decrease which is very uh good um yeah so to wrap up uh yeah this part so yeah we are we having many many tasks

in the LLM part um and we use uh um we start simple with a prompt then we try to optimize it and and uh replace it with in-house models now the question is about evaluation how do we evaluate each task offline and also online so

part not just dialogue we have many other tasks and for all the tasks we are using similar building blocks so if we have question answering in dialogue we can have also question answering in a property page in booking and not just in the dialogue we have text generation

tasks we have text summarization so we have many many tasks and we want to be able to to have a a method to evaluate our LLM outputs so yeah we have all what we saw right now we have performance evaluation like I just show you about

Precision recall, but how we can know about the quality of our outputs? So yeah, now it's more about the quality. So for that, I will explain about three main metrics: factual accuracy, context relevance, and answer relevance. And the idea of having this quality

metrics on our outputs is to make sure we are actually accurate, we are relevant, our outputs are clear and readable. And this is how we build trust with our customer and we meet the product needs and the performance metrics. Um, so just to explain about

these three metrics as just one example. So, uh, let's take, uh, if we have a question and we send it to the LLM, and we get the answer from the LLM. Also, we have a document which is the context that we give to the LLM to be able to

answer this question. So, context relevance is about, uh, given this context, this document, and the user question, do we have actually an answer, um, to these questions in the document? Are we able to answer it? So, this is about context relevance. Does our

context, our document, is relevant to the user question? Answer relevancy is about answer and question. So, given the question and the answer, are we actually answering the user question or are we answering something else? Um, is the answer relevant to the question? But

it can be relevant, but it can be relevant, but it cannot be accurate, right? Like, we can answer something which is relevant to what the user asked for, but maybe it's a hallucination, it's not correct. So, this is why we have factual accuracy, which given the answer and the

document is the answer to true, actually. True is it actually mentioned in the document? Um, so that's how we, uh, we're building LLMs as evaluators, and the process to do that is first offline. We have a minimal data set. We have different prompts to judge LLMs, of LLMs.

evaluators. We are, um, um, uh, having different models. We are trying different models, uh, and then we're comparing cost versus performance of the, of these judge LLMs because we have some label data. Um, and then we are doing some ping pong between the, this, so we are

optimizing the prompt, checking the cost and performance. And once we are, we have, we lock them. We have a prompt that we know it's a good judge, and we know which model to use for this, for this judge, uh, we can move from offline evaluation to

online evaluation. And then what we are doing is we actually take production data, sample it, and run these evaluators on these, uh, outputs. And we are using aggregation to log and alert if we see some, um, drift in factual accuracy. We have some increase in performance of the

factual accuracy. So we want to know that and to, to trigger an alert for that. So we need, we know we need to know that, we need to improve something in the system. Um, yeah, I think that's it. Um, yeah, so for example, for context, relevance is relevant

for dialogue and question answering, answer relevance is relevant for different tasks. Factual accuracy is relevant for actually all the tasks that we have, and that's how we define for each task what evaluators we want to have. Yeah, that's it for my part. If you have some

questions thanks thanks so much for walking us through that run and and um let me go through maybe a little bit of questions before kind of you hop into I know the the whole next set of questions here let me see if I can pull some from from the customer uh sorry from from the

chat here so we have some folks asking um are so your evaluations that you're doing um is it do you do both online and offline are you actually running the evaluations um online data yeah so this is actually the last slide I show Han can you go one

above two above yeah um yeah so offline we are doing um one more yeah offline we are doing the process of setting the right prompt of what will be our evaluator our judge for uh each task once we block the the our evaluator and we know what we need um then we we use

this evaluator in online process we don't do it for every uh llm output because it's very costly we sample we do random sampling on our data set uh and then we run evaluation on top we have dashboards that we can see over time what is our factual accuracy what is our

context relevance all of that yep yep yep and I know that's a it's a key part of how you guys do the evaluation workflow um can I ask okay maybe one more question um so maybe this is level R there's there's a lot of different

components of this application there's not just the llm part there's obviously you're going to the recommendation models there was the N service that you have how did you guys um you know this evaluation you're talking about is about specifically that llm component um but how do you guys

also how do you think about evaluations across the entire flow I will let you explain this I think the way that we look at it is that there's different aspects of evaluation so we start from the LLM which is the single component then we'll look at a

task which is like the NLP service so we're looking at question and answer that could be multiple LLMs but then we also could look at an overall task I want to do recommendation that includes calling the skills or the tasks that Moran has in the NLP service but adding

to that internal services and components and that's a full solution to the user that is what the user is actually going to see at the end of the day and then we have a dialogue level or a full system level so at the end of the day we kind

of have these different um evaluations or systems or metrics or dashboards and each one of them gives us kind of information about a different aspect of of the problem yeah got it okay um maybe we let you dive in and talk about that entire intend the complex agents I think what you're

about to get into is it's not just one piece to evaluate yeah um okay so diving into the other two systems when we started working on the trip planner we actually had the recommendation platform which is a centralized platform for serving all of the recommendations cross booking and

I'll talk about that first and afterwards I'll talk about the Gen orchestrator because it's actually built on top of the same key aspects and key kind of ideas so what is the recommendation platform if we look at booking at the funnel at the website we

We have many different recommendations that we serve in different areas. We have property recommendations, destination recommendations, the trip planner has attraction recommendations. We might want to even recommend a default airport when you start searching, and we have drop-down recommendations when you start typing for different aspects that we can

recommend. All of these are different verticals and different items that we want to recommend, and you can also see that each recommendation has its own enrichment, maybe an image, maybe some interesting review to the user, but it's personalized. Everybody wants to see different information about these items.

So the platform knows how to actually centralize that and serve the right recommendations and the right metadata to the users. Why is this useful? Where did this come in place in the trip planner? If we look at a conversation, we we actually have a lot of different

types of recommendations that we want to provide the user: popular attractions nearby, properties nearby, destinations. When people have the ability to write their requests, they can actually be very dynamic and very personalized about what they want. They can be very specific. It's not structured. It gives them the ability

to say anything, and we need to be able to either structure that or understand it, and then use our internal data to provide the best recommendations to the users. So if we look at that from like a higher level, we want to be able to do popular recommendations nearby,

recommendations, deals, similar themes, and this requires first of all identifying what is the user's interest, but then also being able to pull the right items from our data sources and from our different models or use an LLM maybe even as a recommendation model, and then

provide those items back to the user and if we look at the playground that we've built so inside our platform before we even started the trip, and we have the ability to visualize these recommendations even before they were shown to users or put into production, so

product managers could actually look at these recommendations and understand what they were going to serve and this became very helpful when we were building the trip planner so different vertical, different recommendations were actually being able to kind of understand them and then put them into the trip planner when we went into

production. Okay, so that's the recommendation platform and that was the base of the AI trip planner. Once the NLP had the judge and the ask and collected all the information, we then had to structure that information and use it calling our service, the recommendation platform. Think about it as another agent

or another skill that we were using in order to provide the recommendations. So now if I look at it as an orchestrator, the orchestrator has to be this one centralized system that knows how to take different use cases and different needs and orchestrate

these calls both to the NLP service which does LLM behind the scenes but also to internal services. If I want to pull free text filters, I need to know how to connect to the search service in Booking. If I want to do review summarization, I need to be able to show

that on the property page or somewhere else inside the funnel. So I have to be able to call Booking internal services but also call the LLMs when they're relevant, when they're needed. And if we look at what actually is hiding here behind the scenes, we can see that

we also need to know how to respond correctly so that the front end knows what to show the users. But if we go back to the trip planner, we might want to show a template, we might want to show text results like textual information. Maybe we want to show social bubbles so we

want to have people be able to click into this conversation and be more kind of understanding of what the user is trying to say. And finally, we also want to show our recommendations. We want to bring users into the funnel. Each one of these requires from the orchestrator a

different API, a different response back to the front end, but also we have to fulfill this information in a different way. If we want to bring bubbles now, we have to call the LLM or the NLP service with the right endpoint or the right

skill in order to provide these bubbles. So if we look at a conversation, I'm, I'm a trip planner. How can I help you? And someone says, I want to travel in August. Now we have to trigger the right skills. What do we understand from the conversation? Oh, the user said August.

They talked about when they want to travel, but that's not enough for a recommendation. I can't really provide yet something meaningful to this user. So we're going to have to ask, "Great, where were you thinking of going?" And then the user will say, "Paris." Now I've collected

enough information in order to actually provide a recommendation for Paris in August. So if we look at that, first of all, we need to take that information and format it into something structured: Paris in France and the check-in month is August. Then we need to actually

take that information and turn it into something we can use inside our funnel so "Paris" doesn't mean anything inside booking. I need to actually turn that into an identifier, to something that is meaningful to all of our services and all of our models. Then we can call the

recommendation platform with the right call and the right API in order to return those enriched results. And then we can even call the NLP service again to get some textual information about all these hotels that is most meaningful to this user. If the user said that

they're a family, maybe we want to respond about family-relevant data about these hotels. So on, so forth. If we think of the orchestrator from a higher level, it's responsible for connecting all of these textual responses coming from the LLMs and from the NLP to booking services and

then kind of putting that together into a single response back to the user. How do we do that? Here's an example. So I was talking about the location "Paris, France". If you search booking.com and you start typing "Paris", you'll actually get a drop down of different

locations. Each one of those behind it has an identifier, a latitude, a longitude, a type of location that we understand. That service is something we use and we actually bring to the table when we want to do the structured API to booking identifiers. So this is just one example

of many services that we already have inside booking that we could leverage when we were building the trip planner in order to provide this structured data set so that we can call a recommendation platform which already had all our models and then bring the recommendations to the table which

helped us build the system much faster and if I look at the Dash rep in because again we want to talk about the evaluation and the process that we were doing I couldn't bring the real numbers here but here's just an example of real information that we have inside our eyes

that we use in order to kind of understand what is happening behind the scenes whether it's a detailed information per row so each recommendation what image did we show the user did the user click on it did we get feedback how many messages do the

conversation have and we can also look at a dashboard of what is the number of dialogues we have per day what are the number of dialogues we had in the last month how does this look over time how does this look with NDCG or with ranking

and so on and we can also get alerts so these are like real examples of alerts that we have in the system providing us real-time information that we then can say oh okay we have a problem in production let's try to understand what's going on so we actually have a

Exploring Booking.com's Travel Agent

Arize AI is the watcher, troubleshooter and the guardrail on deployed AI.

