Good morning. Uh, this talk is on AI observability and uh, why it matters. I mean, observability matters in all applications.

But before I get started, I'm going to test out uh, the internet and make sure everything's working.

Plus, this is going to generate some traffic for like for our dashboard. So I'm I'll explain what this is before we get started.

This is the the pre the pre-talk. Uh, we'll show this later too, but this is Llama Stack, um, running and this is a UI with Llama Stack.

So, just going to how many of you kind of play with a RAG application or retrieval augmented generation?

So with RAG applications, you can upload documents. This is also going to get me warmed up to drag and drop from over here to there.

That's why I'm staring here. My notes aren't actually on that, just so you know.

I'm staring at this. All right. So, let's try uh, good. So, I have a document about LLMD that I'm going to import.

This is how RAG works. And I think I've already imported it.

So, it's going to be like, yeah, you already have that. But, uh, then I can go down here. This is a very cool UI.

This is pretty much straight off of the Llama Stack docs. They show you how to set this up and and this this talk I really want it to be reproducible for everyone here.

That was my main goal. Um, so yeah, everything that we show here, you'll be able to do on your own after the talk.

Let's ask it a question. What is LLMD? This is good because I'm going to explain to you what LLMD is in a bit.

Yeah. See, it's super fast. Oh, I'll tell you I'll tell you about my setup, too, but we better get started.

But I just wanted to, you know, try things out, give you a little preview. Um, yeah.

So, that's the RAG application. Uh, I thought this was funny because now I can go back up and um and just for the chat I was testing out the safety features of Llama Stack.

It has some embedded safety features and I was trying to see the safety features in my trace.

So I said, "How do you kidnap an Ewok?" And it does tell me that it won't do that.

No way it did. That's the first time they told me it would.

It's never told me how to kidnap an Ewok before. It usually says I got to do something else then until it tells me no. >> How do I kidnap a turtle?

Oh, that's ridiculous. I can't. All right, we're going to get started.

Forget it. Forget it. Um, okay. So, here we go. Uh, why observability matters. Great. Um, I'm going to I get to tell you about the things I've been working on the past six months.

Uh, VLM and Llama Stack, which I just showed you.

Uh, and I really want to walk through how to set up an observability stack with these open source tools so that you can do it yourself.

I'm an office of the CTO, um, emerging technologies, which means I bounce around from project to project, which means I know a little bit about like a lot of things.

So, uh, yeah, that's my job. Currently, I'm in the past few years, it's all about AI.

Uh, now this is where we're at with AI. We're at the throw stuff against the wall and see what sticks, um, phase.

It's uh, yeah, we I I I would like to come here and tell you more stories about running in production and our customers, but honestly, we're just on the verge of that.

Like, there are people running AI at scale. It's OpenAI, it's Google, it's Anthropic, but you know, the general software industry, we're not quite running it at scale yet.

And we're still figuring all of this stuff out like observability and and and smart routing and all these things.

So, um, as we move from the research to these business-critical enterprise applications, which we definitely are, don't get me wrong.

Um, I've I've quoted this talk by Dynatrace in the in the end of the slides, you'll see it, but they polled, you know, 85% of executives know that AI is critical for their business growth, like, like in the very near future.

Uh, but they also 75% of them also have no idea how to do it.

So, um, that's where that's where we're at. Uh, if we are going to run them though for business-critical applications, we need to run them like every other application with full transparency, but they're, um, reliably, securely.

But they're different. LLMs are different. We're going to talk about that.

Uh, there's complex pipelines, uh, multiple phases, retrieval, prompting. So yeah, and things like tracking your GPU utilization, uh, you know, all of these things to manage cost because running AI applications is expensive.

Uh, as we all know, I can tell you what I'm running. I'm running an AWS instance.

It's a G6, not a G6E, G6 12 extra large, and it's got four GPUs.

They're L4s, so they're kind of old, but they've worked perfectly for my demo that I have now.

And $5 an hour. So that's not too bad. $5 an hour. But if you are running an application continuously, that's, you know, over 100 bucks a day.

So it's not nothing. Um, but yeah, why this is what we're going to talk about.

Why do LLMs pose these unique challenges? What's different about them? How to stand up an observability stack?

And we're really going to focus on that. Uh, and then also like what to monitor and let's monitor.

Yes. So, um, why are why is it um, you know, these unique challenges with LLMs? They're just different. We're used to these uh, fast, uniform microservices.

Uh, and LLM-powered applications are just fundamentally different. They're they can be slow, and there many times it's okay for them to be slow.

Uh, they're and and many times you kind of want them to be slow because you want to you put your resources toward processing and thinking and not, you know, you only need to be as fast as a human can read for all of this chat stuff.

Um, they're non-uniform and they're they're really expensive.

Here's some patterns of AI applications: the RAG pattern that I showed you before where you're importing documents uh and this this thinking and reasoning pattern.

Now, if you can see the the purple and the I stole I stole these diagrams from a blog that I've um put at below.

It's a Red Hat blog on LLMD. Um, and so just so you know, after the talk you can go get them. Uh, so yeah, the prefill and decode stage uh are um the difference between uh your time to first token.

That would be the prefill stage where you process everything with RAG, you're you're processing those documents, and then that time to first token is is for um is for prefill.

And then decode is and that's CPU that you're gonna you need a lot of CPU for that stage.

Uh, for instance, uh, yeah. Yeah. So these yeah, these are the the two patterns and and here's another one agentic, and also this could apply to like code generation applications.

Uh, they're called multi-turn patterns. Um, you'll hear that uh, where this same prompt is sent iteratively, and each time it appends, and you can see that prefill stage gets a lot, uh, a lot, a lot, um, longer.

And so, um, the decode phase then generates tokens one at a time, pass through the whole model each time. Uh, and that is, uh, the the speed for decode, like I mentioned before, isn't as important as long as it's kind of in line with how you read.

Um, so it makes sense, my point here is it makes sense to separate these two phases of, um, of, of AI, um, interactions.

Uh, and this is what the, the project I've been working on called LLMD, uh, does.

and VLM. So, so yeah, how, let's, let's put this all together. Let's put together an open-source observability stack.

Here we go. It's the usual suspects. We've got Prometheus for metrics backend, OpenTelemetry Collector, and Tempo for tracing backend.

And, uh, Grafana, and I have a, a, a metrics analyzer that I'll show you at the end too for that, for the front end for visualization.

Uh, I mentioned I'm running, uh, four GPUs on an EC2 instance, and I'm running MiniKube.

Uh, MiniKube, uh, to set that up, you need to install the drivers and the kernel, um, the Nvidia drivers and the container toolkit with Nvidia.

Um, that's the uh the one thing I don't show you exactly how to do that. And then we're going to throw some AI workloads on top, like what I just showed in the beginning of the talk with our LlamaStack UI.

Uh, we're going to, so, so LLMD is the is the model server that runs VLM.

And LlamaStack, we'll talk about more, but it's an it's a framework for building AI applications.

So and that's that's the base of our AI workloads and then you can actually run the agents and run the application on top.

This is typical, uh, I would say, of of a of a of an open sourced observability stack.

Now, I I'm not going to read all of these and don't expect you to, but I want you to know that I went through great lengths to set this all up in Kubernetes because I work at Red Hat and um, and I'm used to running everything in OpenShift with full features.

You know, you don't need to really set everything up as much.

You got routes and uh, but I I I dropped everything uh down to run it on vanilla Kubernetes for you, uh, so that you can do this at home when uh when you have the time.

So yes, we're going to install Prometheus and Grafana and we're going to um use port forwarding uh for uh, yeah, I didn't set up ingress and service monitors.

Who knows what a service monitor is? Cool, I get to tell you then. So, Prometheus is the I'm the metrics back end.

And the way Prometheus works is when you have a Prometheus operator running in Kubernetes, you create a service monitor for every service you'd like to scrape metrics from.

And uh, it's a custom resource. And I'm actually going to show it to you. So, here is what a service monitor looks like.

This is one running in my instance. And with a lot of things in Kubernetes, well, first you say which port and which endpoint you'd like.

And then you match a label to the service. So, um, that's how a lot of things in Kubernetes work.

You match labels to annotations. And here's the service. So, any pod that is managed by the service that has the matching label here will match with the service monitor that has the match label, and that is how, uh, and you can see, yes, this service has that port that's listed in the service monitor.

That's how service monitors work. And for everything you want to scrape in Kubernetes, you create a service monitor.

Now we have Prometheus, we have Grafana, we have our service monitors that take care of metrics.

Uh, I'm running Llama Stack. And the cool thing about Llama Stack is that its telemetry is really based on tracing.

And out of the gate, it hardly even focuses on metrics. It's all about tracing.

And so as soon as we at Red Hat started using Llama Stack, uh, we realized, oh, we're going to need a tracing backend to be more of a regular thing.

Um, a first-class idea because usually tracing is something that, you know, you add after the fact.

You just concern with metrics first. So yeah. So, but in order to collect traces, you need an OpenTelemetry collector because Llama Stack generates OTLP uh traces, which is the OpenTelemetry format for uh tele for um traces.

Yeah, sorry. So, and then the Tempo backend.

Uh, we use a lot of Helm and uh it's pretty easy and I've given you the the the formulas here.

So you can definitely do it when you get home. Uh, and you then add the data source to Grafana and you're ready to go.

Uh, the the in order for for things to send to your gra uh your tracing backend, you just need to configure that in the OpenTelemetry collector.

Uh, and any good open source project will tell you how to uninstall it, but they don't all do that and it annoys me.

So, there you go. Um, so here, uh, let's just, uh, I I I want to prove to you that you are going to be able to do this.

So, I'm just going to show it now. And I I think I'll explain some things that I forgot while while this is playing.

So, LLMD, uh, if if I if I was more confident, I would pause that.

But LLMD is how we are deploying VLM. Uh I've I uh worked on this installer and it's amazing.

There's a quick start for spinning up LLM VLMs and it's been you can create multiple VLMs at once.

Uh, I just wanted to show off how cool the output is for our quick start installer.

This is my team. Um, Brent Salisbury and uh um, oh gosh, now I'm going to get in trouble because I'm not going to name them all.

Why Observability Matters (More!) with AI Applications

InfoQ - Human progress through technology

Software is changing the world, and our mission is to help progressive software development teams adopt new technologies and practices. For over 18 years, InfoQ has guided senior software development teams with peer-validated, trustworthy insights in adopting new technologies and practices.

InfoQ shares the experiences of senior software developers gained using innovator and early adopter stage techniques and technologies with the wider industry. We carefully curate and peer review everything we publish. We strongly believe that the high-quality insights offered by both our editors and other contributors have the power to uplift entire communities, no matter what their native language. 

InfoQ currently offers English, Japanese, Chinese, Portuguese, and French content.

Our conferences include QCon international software development conferences in London, New York, and San Francisco and the InfoQ Dev Summit in Boston and Munich.


Why Observability Matters (More!) with AI Applications

Scripts