Hi everyone, I'm Fabian, and I'm a member of the technical staff at OpenAI on their applied observability team. I'm super excited to be here and share with you how we're using open-source technology to process logs at scale and some of the interesting things that we've found.

So, it's only been three years since the original ChatGPT release, and I think it's fair to say that AI has already completely changed the way that we work and the way that we access information.

With GPT-5, the model is not just generating text. It's seeing and creating images, writing and running code, and orchestrating increasingly complex tool chains.

And each step up in capability doesn't just mean more compute. It means more Kubernetes clusters, more user traffic, and orders of magnitude more telemetry to keep it all running and ensure that we're successful in our mission to make AGI beneficial to all of humanity.

Our role on applied observability is to make sure that every engineer at OpenAI has the tools that they need to prevent SEs hopefully before they happen, but also to effectively fight them when they do.

And that means ingesting absolutely massive amounts of telemetry and making it fast, queriable, and actionable at scale.

So to set the scene for you, I'm first going to share with you the high-level architecture of what our logging solution looks like.

Then I'll share the unique challenges that we're trying to meet and how those business problems force us to look for performance wins.

And lastly, I'll share the Eureka moment that saved us a ton of compute and what I think is the most important thing to take away from this experience.

So the ingestion pipeline starts with Fluent Bit.

We run Fluent Bit on every Kubernetes node in our fleet, and it's responsible for ingesting log files, performing any sampling, enriching those logs with metadata that comes from Kubernetes, formatting them into the right schema, and then routing them to the appropriate data stores that those logs can be queried.

In the aggregate, we emit just shy of 10 petabytes of logs per day using this architecture.

And since Fluent Bit runs on every node, its performance is critical. A Fluent Bit process that exceeds its CPU quota and gets throttled could end up losing logs or worse, it could degrade the performance of a different application.

So, we'd prefer it to just use as little CPU as possible to get the job done.

So, if you follow the news, I'm sure you've heard about our absolutely insatiable appetite for GPUs.

But what gets talked about less is that we also need a lot of CPUs, too. We know we need more capacity to meet our ambitious goals, but not all the capacity that we ask for can ever be acquired.

So, this necessitates us to optimize our compute as well as horizontally scale it. So if we can return a CPU to every node, then maybe that's one more microservice that we can fit into a given host.

So we call this the DaemonSet problem, and it makes it super hard for us to justify creating a new one because then any resources that it requires slightly reduces the resources that every node in our fleet contributes back to the cluster.

Um, ideally we'd have just far fewer DaemonSets in our environment and the ones that we do have would use far fewer resources.

So through this lens, we set out to optimize Fluent Bit and make it just as lean as possible.

And luckily, there's pretty great tooling these days for figuring out where CPU cycles are going. Um, Perf is Perf is a powerful tool for doing these kinds of investigations, and it works with any language, and I like to start these investigations with a hypothesis.

So my hypothesis going in was that Fluent Bit would probably be spending most of its time doing expensive string processing because we do all that enrichment, filtering, uh, log processing inside of Lua, and maybe that would cause a bunch of load as Fluent Bit had to do all those computations, memory allocations, that kind of thing.

But let's let's see what the report says.

This is the output of the Perf report command, and my theory was totally wrong. If you look at the symbol that's highlighted, consuming about 35% of this thread's time, it's called `fstat64`, which is used to return information about a file's various properties, such as including size.

And this is a super surprising result that really didn't match my expectations going in. In essence, this is saying that the biggest use of Fluent Bit's CPU time is spent figuring out how large log files are before reading them.

So there's two places that use `fstat` in the tail plugin. The first is in the periodic refresh interval, which we've configured to run a stat on every file every 5 seconds.

And that's frequent, but if we only have a few dozen pods per node, it's just it's never going to be high enough in frequency to justify that 35% of CPU time.

Um, so something else has to be driving it. And the second place that stats will happen is via inotify.

Uh, it's enabled by default and it reduces Fluent Bit's need to do that periodic uh, stat-based polling since the kernel will just let us know when a file has been written to.

And so this PR, it's a one-liner. We just we turn inotify off and let's see what happens.

Uh, rolled this out to a single test cluster and the results speak for themselves.

We have a new load pattern here that uses about half as much CPU while doing exactly the same work. But why why is this happening?

This is actually a really surprising result. We have to think about it from Fluent Bit's perspective. It's not just enough to know that a write has happened.

We also have to know how much data has been written. And this information is useful because we have to feed it back into our state tracking database regarding like where we are reading in the file.

So in response to the inotify event, Fluent Bit will issue that fstat call immediately that retrieves that information and then immediately read the file's content for new data.

And if a process is continually emitting new logs line by line, then Fluent Bit is going to race that and continue to run stat for every time that that write happens.

And that's going to burn a ton of unnecessary compute, as it was in our environment. So, we rolled out this change to all of production, and the results held up pretty nicely.

It actually ended up returning about 30,000 CPU cores worth of capacity back to the Kubernetes cluster.

And this is a result that's going to just continue to compound over time as we bring more compute online.

Not all these hosts are busy at the same time or have the same services deployed with the same logging characteristics. But across the board, the result is that 50% drop in CPU to our heaviest loaded Fluent Bit pods.

And at the time we rolled it out, about 30,000 CPU cores at peak.

This was one of a series of performance enhancements that we've made to try to improve our use of Fluent Bit over time.

Although this was probably the most effective in terms of lines written to payoff. We're working to propose a patch upstream so that you can specify like a minimum interval or something like that and see the same kind of results in your environments, maybe without having to disable `inotify` too.

So, as an industry, in the last decade, I think we've gotten much better at building highly scalable distributed systems, and that's definitely a good thing.

At OpenAI, we've been able to leverage that knowledge and the incredible open source ecosystem to reach a level of scale that in the past would have maybe required a decade to prepare for and had involved a lot more proprietary software.

It's a testament to the power of open source that when ChatGPT took off, we were able to meet that moment.

Open source software gives all of us the ability to focus on our core business problems and not worry so much about the other stuff.

Still, just because our community's gotten so good at scaling things horizontally doesn't mean that there's not value in breaking out your profiler of choice and seeing what's really happening under the hood and on your hardware.

Sometimes you might just find something surprising. And truthfully, doing it these days is easier than ever.

You don't need to be a C programmer to ask Chat to help you run Perf for the first time, to get the command line arguments right, or even to interpret the results.

You just need 15 minutes, root, and a little bit of patience. So my closing message is this.

If you're into this kind of thing, if you think this is cool, take a few minutes out of your day and profile some of the open source software that you use at your company under load, and let's see if we can find anything else surprising and work to make open software or open source software a little bit better for everybody.

Keynote: How One Line of Code Freed 30,000 CPU Cores: Deep-Diving Fluent Bit at Petabyte... F. Ponce

To provide educational and informative content on cloud native computing, which uses an open source software stack to deploy applications as microservices, packaging each part into its own container, and dynamically orchestrating those containers to optimize resource utilization. Educational content on CNCF projects, like Kubernetes and Prometheus, will also be provided.

Keynote: How One Line of Code Freed 30,000 CPU Cores: Deep-Diving Fluent Bit at Petabyte... F. Ponce

Scripts