Kubray: Managing Ray Clusters and LLM Lifecycle on Kubernetes

Kubray, a Kubernetes operator, manages Ray clusters and their applications, offering three core components for different workloads.

### Kubray: The Kubernetes Operator for Managing Ray Clusters and Applications
* 🚀 Kubray is a Kubernetes operator that manages the entire lifecycle of Ray clusters and their associated applications on Kubernetes, allowing data and ML scientists to **focus on experiments and computation** while infrastructure engineers handle **deployment and integration** with Kubernetes ecosystem tools.
* 🎯 Ray excels at **large-scale data computation, especially for machine learning tasks**, while Kubray seamlessly integrates Ray into the **Kubernetes ecosystem**.
* 📦 Kubray offers three core components for different workloads: the **Ray Cluster Custom Resource** for long-running or lightweight ephemeral clusters, the **Ray Job Custom Resource** for ad hoc or scheduled workloads, and the **Ray Service Custom Resource** for services where uptime is critical.
* 🧠 Both Ray and Kubernetes serve as resource orchestrators, but they focus on different aspects: **Ray is all about computation**, while **Kubernetes is geared towards deployment**.
* 🤖 For large language models (LLMs), the transition from CPU to GPU inference and the increase in computational costs have made **serving LLMs more expensive and important**.
* 🎉 Kubray is uniquely positioned as an **optimal solution for deploying and managing LLMs on Kubernetes**, especially for **LLM serving costs**.
* 📈 Kubray has auto-scaling capabilities for online serving workloads and supports **heterogeneous computer resources** like different types of **GPUs and CPUs**.
* 🔗 The Kubray 1.0 release is generally available, with a strong community, significant user growth, and success stories like **Niantic reducing game processing costs by 60%**.

foreign a software engineer at any skill with me today is Kai shinchen kashin you want to introduce oh hi I'm kaishin and I'm a software engineering and scale and the focus here on kilbrey thanks so much for joining us today um we're delighted to talk to you about kubrey which is a kubernetes operator for deploying and managing Ray applications today's agenda will cover the core components of kubrey including the ray cluster custom resource rayjob custom resource and Race Service custom resource we'll also dive into some real world use cases and even walk you through a demo of how Cuba can handle the whole llm lifecycle from fine tuning to serving and more finally we'll discuss some Community News including collaborations and testimonials so what exactly is kubrey kubrey is a kubernetes operator that manages the entire life cycle of Ray clusters and their Associated applications on kubernetes to paraphrase the kubernetes docs operators are software extensions to kubernetes that make use of custom

resources to manage applications and their components. In this case, Ray clusters, Ray jobs, and Ray services. And I'll talk more about those later. Ray excels at large-scale data computation, especially for machine learning tasks, while Kubray focuses on seamlessly integrating Ray into the Kubernetes ecosystem. What you can see from this diagram is that Ray provides you with two ways to manage the Ray infrastructure: either directly through the Ray VM cluster launchers or through Kubray for Kubernetes deployments. In this talk, we're focusing on Kubray. Why is Kubray the best solution for running Ray on Kubernetes? Well, Kubray enables data and ML scientists to focus on what they do best: experiments and computation. Meanwhile, infrastructure engineers can concentrate on the deployment side and on integrating Cube Ray with Kubernetes ecosystem tools like Prometheus, Grafana, Nginx, and others. The details of the arrows here are not so important. If you're familiar with Kubernetes operators already, it's just a standard pattern for operators. Some the summary is that both Ray and

Kubernetes serves as resource orchestrators, but they focus on different aspects. Ray is all about computation, handling tasks, and actors, while Kubernetes is geared towards deployment, managing pods as its scheduling units. So, the main takeaway here is that this allows for the separation of responsibilities between data and ML scientists and infrastructure engineers. Kubrey offers three core components for different workloads. For the first use case, imagine you want to have a long-running cluster to submit multiple workloads to, possibly from multiple users, or perhaps you want to spin up a temporary cluster for debugging, writing unit tests, or prototyping. For these use cases, you can use the Ray Cluster Custom Resource. At the top here, the Ray Cluster Custom Resource manages the lifecycle of a Ray cluster. It supports auto-scaling, and it also supports GCS fault tolerance if the Ray head node goes down. And I'll talk more about that later. For the second use case, imagine you have ad hoc workloads where you don't know when they'll be submitted, but you don't want to pay the cost of

having a live cluster up and running all the time. For a case like this, you can use the ray job custom resource. Ray job is essentially two components: a ray cluster plus a job. It automatically creates a ray cluster and submits a job when the cluster is ready using the ray job API. Best of all, you can recycle the ray cluster automatically after the job completes. Finally, suppose you're responsible for a service, so uptime and latency are prime concerns for you. For this use case, you can use the ray service custom resource. It not only creates a ray cluster but also ensures zero downtime upgrades and high availability, which are features that we'll dive into in the next few slides. To sum it up, you can use the ray cluster custom resource for both long-running clusters or lightweight ephemeral clusters. You can use the ray job custom resource for ad hoc or scheduled workloads. And you can use the ray service custom resource when uptime is paramount. And for those who may be less familiar, a custom resource is simply an extension of the Kubernetes API that can be used.

alongside the built-in resources that you're familiar with, such as pods. For example, like other resources, you declare the desired state of the resource, and the Kubrey operator keeps the current state of Kubernetes objects in sync with the declared desired state. So, for each of these three custom resources, we're going to dive into the details of how that's accomplished. We'll start with the Ray cluster custom resource definition. As mentioned earlier, Kubrey follows the classic Kubernetes operator pattern for managing Ray clusters. This slide shows the architecture of the Ray cluster custom resource definition. As a user, you just need to create a Ray cluster custom resource. The Kubrey operator then automatically takes care of the creation, deletion, and updating of any Kubernetes resources that it needs to spin up the Ray head and worker pods. The idea is to let you focus on the Ray cluster configuration while Kubrey takes care of the Kubernetes aspects. One key feature here is that Kubrey integrates seamlessly with the Ray Auto Scaler. This allows your Ray clusters to dynamically scale based on the workload.

you can add GPU pods when they're needed, and remove them when they're not. So here we have an array cluster that's going back and forth between these two states. In the minimal state, it's just a head node which only has a CPU. But if GPUs are needed for a workload, that same cluster can scale up. Using this, you get both performance and cost efficiency. Now let's talk about Ray job, a custom resource that's particularly beneficial for those concerned about Cloud resource costs. You create a Ray job, and then the Kubrey operator springs into action, creating, updating, and deleting the Kubernetes resources. And one great thing about this is, as we mentioned previously, once the job is done, it will automatically delete the Ray cluster, conserving your valuable Cloud resources. And the deletion, of course, is an optional feature. You can disable it in case you'd rather leave the cluster up and running for postmortem debugging. Finally, let's talk about the Ray service custom resource. Before going into the architecture diagram for the Ray service, let's just

describe two of the key stability features that Ray service brings to the table. First up is zero downtime cluster changes. Ray Service automatically detects both unhealthy cluster states and user config changes that would necessitate a new cluster. In response, it spins up a new Ray cluster, shifts the traffic, and finally deletes the old Ray cluster. And what about high availability and GCS fault tolerance? The GCS, or Global Control Service, is a core component of the Ray head node that manages cluster-level metadata. But even if the Ray head pod goes down, Ray Service ensures that user traffic continues to flow to worker pods uninterrupted while the new head pod starts back up. This makes it highly available and fault-tolerant. The architecture diagram for Ray Service is pretty similar to the ones we've seen before. The user creates Ray Service custom resource. The operator monitors the service and updates Ray Service applications in place. The Kubeflow operator takes care of creating, updating, deleting the Kubernetes resources just like it does for Ray cluster and Ray job. And note that in the case of the zero

downtime feature that I mentioned in the previous Slide. The race service might be managing two Ray clusters, one of them pending, one of them active as indicated in the picture. Finally, with the setup, the user maybe a different user from the one who started the race service can directly query the race serve application, which is exposed as a Kubernetes service. In summary, Kubray is the go-to solution for managing Ray clusters and applications on Kubernetes. It saves time, reduces costs, and enhances stability. So far, we've talked about what Kubray can do for general Ray clusters and applications, but let's focus on an area that's increasingly important: large language models or LLMs. Why is this different from traditional ML workloads? There's a transition from CPU to GPU inference. The computational costs increase dramatically. There's also a change in the life cycle. Instead of starting with data and using it to train a model for specific tasks, companies and startups can take a pre-trained foundational model, which may have been too costly to train themselves.

from scratch, and then they can fine-tune it for their use case. This shift has resulted in an increasing number of LLM application developers. For LLM's distributed computation is a must-have. We think Cube Ray is uniquely positioned as an optimal solution for deploying and managing LLMs on Kubernetes. We'll share insights on how you can manage all stages of the lifecycle of LM deployments using Kubray. And with that said, I'll now hand it over to Kaishin to tell you more. Thanks, Asha. I'm Kaishin. Today I will talk about LM with Kubray. The first is that we will compare the difference between the traditional ML model lifecycle and the LM model lifecycle. For the traditional ML lifecycle, it consists of several stages, including data preparation, model training, tuning, and the model serving. And typically, the most expensive part is the model training because it requires a lot of GPU. And therefore, the serving part maybe it is not that expensive because the model is not large, so you can use single or a

small GPU for the inference, and as I know if your workload is not very sensitive to the latency or some company also use CPU for the inference to set the cost, uh, and our acceleration is that the performance bottleneck for the model serving for a traditional ML is about the computation, not memory, and then, uh, for the LM model life cycle, it is a bit different. The main difference is that still then P sell the company that's a trend layer Foundation model from scratch because prescribing a large language model is pretty expensive and the inmate Tech, uh, several million dollars, and the only big Tech can afford it like The Meta like OpenAI, yeah. So based on our observation, most users use open-source pre-trained large language models like a metas or Lama two like Falcon like vicuna and then use their data set for the fight tuning, yeah, and the fine-tuning is will not, uh, will not cost a lot of money, but for the serving, it's a totally different story because the not because the model becomes very large, for example, like the

number 270b. In most of Industry case, we will not use the consultation, so we will use f316 for each parameter. So it's required 140 gigabytes of GPU memory, so we need multiple and the largest GPU for the inference. So it becomes very memory intensive. So there are a larger of the memory appendation technique, like the continuous batching, like the patch attention from the vln, like a fashion attention, yeah. And we have any scale published a lot of blogs, a lot about that. So if you are interested, you can take a look at it, yeah. So our problem is that the serving becomes much more expensive and important than before, right? The ml industry ml apps industry move from the training to the serving, yeah. And for the Aon with qbrew can match the end to end Ln life cycle on kubernetes, and we think that the cube and array are the best solution for large language model and kubernetes, especially for large language model serving cost. The first is that I should introduce.

that we have the auto-scaling because the online serving the workload is not it's hard to predict so we require the auto-scaling to add our data our computer resource based on the dynamic load to step the cost. The second is the heterogeneous because currently it's not it's not that you have money so you can get a GPU a lot of times that you have money but you cannot get GPU especially for some high-end GPU like a100 or h100 so supportive support the heterogeneous computer resource are very important like different types of GPU like GPUs or CPUs and then we will demo the end-to-end a large language model lifecycle with QB. Yeah this is our architecture of our demo the first is that we will do their processing with the recursor Jupiter notebook and the auto-scaling and then after we finish the data processing we will create a region for the phytuning because the workload the resource Computing resource requirement for the fine-tuning is is able to predict so we decided to use rate drop to several course.

and then after the fire tuning, it will upload the model checkpoint to the account storage. And then step three, we can use the ray service to serve our fine to large language model. And the first, therefore, we will build an LM application with the lension and the gradual to showcase the application based on our endpoints served by Race Service. Yeah, for the prototyping side, uh, in this demo, we will not cover this part, but, uh, but this is a recommendation architecture. Uh, the first is that we typically recommend user to run a Jupiter notebook as a head pass cycle container and land, uh, user. We also, in, uh, recommend user to enable the auto scaler, and the auto scaler will inject a cycle container into the headspot. And for the, and the second, for a second, we will fine-tuning with the ray job, and in this demo, we find you a llama27b on the math data set. And in the demo, we use a raychen and a disk bit. And then we have the 16 Nvidia A10 GPUs and a 24 gigabyte RAM for per GPU.

and we only find you for one Epoch, and it takes 22 minutes. Yeah, analysis our demo video. The first is that we install a qbre operator, yeah, and the ones for it should be ready, right? And then creates a great job for fine-tuning, yeah, and check the custom resource, yeah, and we watch it apart, and it one has node and the 16 GPU working nodes, and then we watch the monitor, the fine-tuning job progress, and then after 20 minutes, right, it's the state, and you can see the checkpoint here, yeah, and then because this is for demo purpose, we didn't configure the auto-detecting Direction, so after this is by ourselves, and then after we finish the phytonin, we can serve the, we can serve the fighting model with Race Service, and the ray Ln, and the real M is a previously named every it is a tool that can help you to serve your large language model in Array, and it provides open AI-compatible API, so it's pretty easy to use the open air Python package to use the, to use the.

endpoint Leicester by Ray service and this is the demo. What should you do last time oh here we go yeah. We have already have an operator that in our cluster so we create a red service yeah and then we watch our part. And in the raycaster it will have one head no and the one GPU work node. And we will serve law serve a large language model on the GPU node. And then it has a it is ready so we just try to pull forwarding and then and then use the curl request to test it yeah and we got a response cool and then uh. The last step is that we have a endpoint let's start by our array service with our fine tune llama27b model. So the last step is that I want to build an LM application with Lane Chen and the gradual and here we use graduation for to build a web UI and use the engine to handle the i o between the race service endpoints and our users inputs and output right. And because this one is a RFI 2 model is fine too with the math data set.

so we will use a math question for the demo, and the lantern has a it will use a prompt canvas. That's as we've shown in the slide, right? This is how we find to our model, so that's why we use this prompt template, yeah. And this is a math problem that we will use. It is the same from our math data set, and the answer is five. It's a pretty simple math question, yeah. And the list is the very short video, yeah, because I have already put forwarding that to my Dev box. So I just run the gradual server that's on my local box, and then you can see this UI, and we use the question that's in the last slide, and the summit should our endpoints, yeah. And the result from our model is not correct because we just find you for one Epoch, but the response is that it has already followed our data says format, so it is better than the best model, no, yeah. And uh, and uh, actually introduced the current status of the qbre, and I introd, I talked about the LM with qbre.

So currently, I want to recapital 2003 for qbre community. Uh, from the last restaurant until today, we have four major release from 230 companies to six more than 600 comments, and we have more than 90 contributors. Uh, like Dimitri from curse, and uh, joshing Wilson, and Chinese from bidance, and RD from Microsoft, and like each time from the, and several folks from the red hat and IBN, right? We have a very, very good community, and thanks for the contribution. And then we have a lot of users blocked like the, like the insta cards, uh, sansara, Spotify, and the draw Dash, and the Google, and uh, we have a massive growth in the number of qbre clusters. We have a 9x growth in the last six months, and the two acts in the August, and uh, in our expectation, we think that this, uh, this month will have another 2x in September. And this is our user's feedback, uh, from the Antics, right? Because I am not sure a lot is this a legal for me to use their.

logo. So I decided to use the Psyduck, which is my favorite Pokemon. Yeah, and uh, Niantic's with Cuberay, and they used to for less game processing, and it's definitely caused by 60 percent, right, which is very fantastic, right? So we are very happy to announce that uh, Cuberay is generally available, right? I think you guys know that in the ion's Keynote, right? And then uh, the currency, the Cuberay 1.0, the first race candy is available, so feel free to try it on. This really extended will be very different from the previous one. For the previous one, we will just iterate with user. For this case, for one to two weeks, but this one, we will iterate maybe for six weeks. So it is a good time if you have any bug, right? It will be fixed very fast, so it's a good time to open some issue, yeah, and thanks for you guys to attending our talk. Thanks.

KubeRay: A Ray cluster management solution on Kubernetes

Welcome to the Anyscale YouTube channel! Join us as we explore Ray, an open-source framework designed to accelerate the development and deployment of machine learning applications. From beginner tutorials to advanced insights, our videos cater to all skill levels. Stay up-to-date with the latest advancements and hear from experts through interviews, panel discussions, and keynotes.

Anyscale

Introducing Kubray: A Kubernetes Operator for Ray Applications

Kubray is a Kubernetes operator that manages the entire life cycle of Ray clusters and their associated applications on Kubernetes, focusing on seamless integration and enabling data and ML scientists to focus on their work.

It offers three core components for different workloads: the Ray Cluster Custom Resource for long-running clusters, the Ray Job Custom Resource for ad hoc workloads, and the Ray Service Custom Resource for serving models.

Kubray's main advantage is the separation of responsibilities between data/ML scientists and infrastructure engineers, allowing for efficient management of Ray on Kubernetes.

Managing Ray Clusters with Custom Resources

Ray job custom resource allows for the automatic creation and deletion of Ray clusters based on job completion, helping to conserve cloud resources.

The Ray cluster custom resource, on the other hand, is suitable for managing both long-running and ephemeral clusters, with seamless integration with the Ray Auto Scaler.

Meanwhile, the Ray service custom resource is ideal for ensuring zero downtime upgrades and high availability, catering to use cases where uptime is crucial.

Ray Service provides zero downtime cluster changes, ensuring uninterrupted traffic shift to a new cluster during unhealthy states or config changes.

It also offers high availability and GCS fault tolerance, allowing user traffic to continue flowing even if the Ray head pod goes down.

The architecture of Ray Service involves the creation of a custom resource, monitoring by the operator, and management of Kubernetes resources, making it a go-to solution for Ray clusters and applications.

The Growing Importance of Large Language Model Serving Cost

Large language model serving is becoming significantly more expensive in the ML industry.

The shift from training to serving has made GPU memory intensive, requiring expensive infrastructure and memory optimization techniques.

To address this, auto-scaling and heterogeneous computer resources are crucial, and Kubernetes is seen as the best solution for managing large language model serving cost.

Building and Deploying a Large Language Model with Ray Service

The process involves fine-tuning the LLM model using Ray, with specific GPU and RAM requirements for the task.

The next step is to serve the fine-tuned model using Ray Service, which provides an open AI-compatible API.

Finally, the plan includes building an LM application with Lane Chen and Gradual for creating a web UI and managing the input-output with Ray Service endpoints.

The QBRE community has experienced significant growth since 2003, with four major releases and over 600 comments from 90+ contributors.

The number of QBRE clusters has seen a 9x increase in the last six months, with an expected 2x growth in the coming month.

User feedback, including from Niantic, has been highly positive, with a 60% improvement in game processing.

0 is now available, featuring an extended six-week iteration period for faster bug fixes.