Kubray: 쿠버네티스에서 레이 클러스터 및 LLM 라이프사이클 관리하기

### Kubray: 레이 클러스터와 애플리케이션 관리를 위한 쿠버네티스 운영자
* 🚀 Kubray는 레이 클러스터와 관련 애플리케이션의 전체 라이프사이클을 **Kubernetes에서 관리**하여 데이터 및 ML 과학자가 실험과 계산에 집중하는 동안 인프라 엔지니어가 배포 및 Kubernetes 에코시스템 도구와의 통합을 처리할 수 있도록 하는 **Kubernetes 운영자**입니다.
* 🎯 Ray는 특히 머신 러닝 작업을 위한 대규모 데이터 연산에 탁월하며, **Kubray는 Ray를 Kubernetes 에코시스템에 원활하게 통합**합니다.
* 📦 장기 실행 또는 경량 임시 클러스터를 위한 **레이 클러스터 커스텀 리소스**, 임시 또는 예약된 워크로드를 위한 **레이 잡 커스텀 리소스**, 가동 시간이 중요한 서비스를 위한 **레이 서비스 커스텀 리소스** 등 다양한 워크로드를 위한 세 가지 핵심 구성 요소를 제공합니다.
* 🧠 레이와 쿠버네티스는 모두 리소스 오케스트레이터 역할을 하지만 서로 다른 측면에 중점을 둡니다: **Ray는 계산에 중점을 두는 반면, Kubernetes는 배포에 중점을 둡니다.**
* 🤖 대규모 언어 모델(LLM)의 경우, **CPU 추론에서 GPU 추론으로의 전환과 계산 비용의 증가**로 인해 LLM을 제공하는 것이 더 비싸고 중요해졌습니다.
* 🎉 Kubray는 특히 LLM 서비스 비용과 관련하여 **Kubernetes에서 LLM을 배포하고 관리하는 데 있어 최적의 솔루션**으로 독보적인 위치를 차지하고 있습니다.
* 📈 Kubray는 온라인 서비스 워크로드를 위한 자동 확장 기능을 갖추고 있으며 **다양한 유형의 GPU 및 CPU**와 같은 이기종 컴퓨터 리소스를 지원합니다.
* 🔗 강력한 커뮤니티, 상당한 사용자 증가, **게임 처리 비용 60% 절감**과 같은 성공 사례와 함께 **Kubray 1.0 릴리스는 일반적으로 사용**할 수 있습니다.

Kubernetes 운영자인 Kubray는 Ray 클러스터와 해당 애플리케이션을 관리하며, 다양한 워크로드를 위한 세 가지 핵심 구성 요소를 제공합니다.

foreign a software engineer at any skill with me today is Kai shinchen kashin you want to introduce oh hi I'm kaishin and I'm a software engineering and scale and the focus here on kilbrey thanks so much for joining us today um we're delighted to talk to you about kubrey which is a kubernetes operator for deploying and managing Ray applications today's agenda will cover the core components of kubrey including the ray cluster custom resource rayjob custom resource and Race Service custom resource we'll also dive into some real world use cases and even walk you through a demo of how Cuba can handle the whole llm lifecycle from fine tuning to serving and more finally we'll discuss some Community News including collaborations and testimonials so what exactly is kubrey kubrey is a kubernetes operator that manages the entire life cycle of Ray clusters and their Associated applications on kubernetes to paraphrase the kubernetes docs operators are software extensions to kubernetes that make use of custom

resources to manage applications and their components. In this case, Ray clusters, Ray jobs, and Ray services. And I'll talk more about those later. Ray excels at large-scale data computation, especially for machine learning tasks, while Kubray focuses on seamlessly integrating Ray into the Kubernetes ecosystem. What you can see from this diagram is that Ray provides you with two ways to manage the Ray infrastructure: either directly through the Ray VM cluster launchers or through Kubray for Kubernetes deployments. In this talk, we're focusing on Kubray. Why is Kubray the best solution for running Ray on Kubernetes? Well, Kubray enables data and ML scientists to focus on what they do best: experiments and computation. Meanwhile, infrastructure engineers can concentrate on the deployment side and on integrating Cube Ray with Kubernetes ecosystem tools like Prometheus, Grafana, Nginx, and others. The details of the arrows here are not so important. If you're familiar with Kubernetes operators already, it's just a standard pattern for operators. Some the summary is that both Ray and

Kubernetes serves as resource orchestrators, but they focus on different aspects. Ray is all about computation, handling tasks, and actors, while Kubernetes is geared towards deployment, managing pods as its scheduling units. So, the main takeaway here is that this allows for the separation of responsibilities between data and ML scientists and infrastructure engineers. Kubrey offers three core components for different workloads. For the first use case, imagine you want to have a long-running cluster to submit multiple workloads to, possibly from multiple users, or perhaps you want to spin up a temporary cluster for debugging, writing unit tests, or prototyping. For these use cases, you can use the Ray Cluster Custom Resource. At the top here, the Ray Cluster Custom Resource manages the lifecycle of a Ray cluster. It supports auto-scaling, and it also supports GCS fault tolerance if the Ray head node goes down. And I'll talk more about that later. For the second use case, imagine you have ad hoc workloads where you don't know when they'll be submitted, but you don't want to pay the cost of

having a live cluster up and running all the time. For a case like this, you can use the ray job custom resource. Ray job is essentially two components: a ray cluster plus a job. It automatically creates a ray cluster and submits a job when the cluster is ready using the ray job API. Best of all, you can recycle the ray cluster automatically after the job completes. Finally, suppose you're responsible for a service, so uptime and latency are prime concerns for you. For this use case, you can use the ray service custom resource. It not only creates a ray cluster but also ensures zero downtime upgrades and high availability, which are features that we'll dive into in the next few slides. To sum it up, you can use the ray cluster custom resource for both long-running clusters or lightweight ephemeral clusters. You can use the ray job custom resource for ad hoc or scheduled workloads. And you can use the ray service custom resource when uptime is paramount. And for those who may be less familiar, a custom resource is simply an extension of the Kubernetes API that can be used.

alongside the built-in resources that you're familiar with, such as pods. For example, like other resources, you declare the desired state of the resource, and the Kubrey operator keeps the current state of Kubernetes objects in sync with the declared desired state. So, for each of these three custom resources, we're going to dive into the details of how that's accomplished. We'll start with the Ray cluster custom resource definition. As mentioned earlier, Kubrey follows the classic Kubernetes operator pattern for managing Ray clusters. This slide shows the architecture of the Ray cluster custom resource definition. As a user, you just need to create a Ray cluster custom resource. The Kubrey operator then automatically takes care of the creation, deletion, and updating of any Kubernetes resources that it needs to spin up the Ray head and worker pods. The idea is to let you focus on the Ray cluster configuration while Kubrey takes care of the Kubernetes aspects. One key feature here is that Kubrey integrates seamlessly with the Ray Auto Scaler. This allows your Ray clusters to dynamically scale based on the workload.

you can add GPU pods when they're needed, and remove them when they're not. So here we have an array cluster that's going back and forth between these two states. In the minimal state, it's just a head node which only has a CPU. But if GPUs are needed for a workload, that same cluster can scale up. Using this, you get both performance and cost efficiency. Now let's talk about Ray job, a custom resource that's particularly beneficial for those concerned about Cloud resource costs. You create a Ray job, and then the Kubrey operator springs into action, creating, updating, and deleting the Kubernetes resources. And one great thing about this is, as we mentioned previously, once the job is done, it will automatically delete the Ray cluster, conserving your valuable Cloud resources. And the deletion, of course, is an optional feature. You can disable it in case you'd rather leave the cluster up and running for postmortem debugging. Finally, let's talk about the Ray service custom resource. Before going into the architecture diagram for the Ray service, let's just

describe two of the key stability features that Ray service brings to the table. First up is zero downtime cluster changes. Ray Service automatically detects both unhealthy cluster states and user config changes that would necessitate a new cluster. In response, it spins up a new Ray cluster, shifts the traffic, and finally deletes the old Ray cluster. And what about high availability and GCS fault tolerance? The GCS, or Global Control Service, is a core component of the Ray head node that manages cluster-level metadata. But even if the Ray head pod goes down, Ray Service ensures that user traffic continues to flow to worker pods uninterrupted while the new head pod starts back up. This makes it highly available and fault-tolerant. The architecture diagram for Ray Service is pretty similar to the ones we've seen before. The user creates Ray Service custom resource. The operator monitors the service and updates Ray Service applications in place. The Kubeflow operator takes care of creating, updating, deleting the Kubernetes resources just like it does for Ray cluster and Ray job. And note that in the case of the zero

downtime feature that I mentioned in the previous Slide. The race service might be managing two Ray clusters, one of them pending, one of them active as indicated in the picture. Finally, with the setup, the user maybe a different user from the one who started the race service can directly query the race serve application, which is exposed as a Kubernetes service. In summary, Kubray is the go-to solution for managing Ray clusters and applications on Kubernetes. It saves time, reduces costs, and enhances stability. So far, we've talked about what Kubray can do for general Ray clusters and applications, but let's focus on an area that's increasingly important: large language models or LLMs. Why is this different from traditional ML workloads? There's a transition from CPU to GPU inference. The computational costs increase dramatically. There's also a change in the life cycle. Instead of starting with data and using it to train a model for specific tasks, companies and startups can take a pre-trained foundational model, which may have been too costly to train themselves.

from scratch, and then they can fine-tune it for their use case. This shift has resulted in an increasing number of LLM application developers. For LLM's distributed computation is a must-have. We think Cube Ray is uniquely positioned as an optimal solution for deploying and managing LLMs on Kubernetes. We'll share insights on how you can manage all stages of the lifecycle of LM deployments using Kubray. And with that said, I'll now hand it over to Kaishin to tell you more. Thanks, Asha. I'm Kaishin. Today I will talk about LM with Kubray. The first is that we will compare the difference between the traditional ML model lifecycle and the LM model lifecycle. For the traditional ML lifecycle, it consists of several stages, including data preparation, model training, tuning, and the model serving. And typically, the most expensive part is the model training because it requires a lot of GPU. And therefore, the serving part maybe it is not that expensive because the model is not large, so you can use single or a

small GPU for the inference, and as I know if your workload is not very sensitive to the latency or some company also use CPU for the inference to set the cost, uh, and our acceleration is that the performance bottleneck for the model serving for a traditional ML is about the computation, not memory, and then, uh, for the LM model life cycle, it is a bit different. The main difference is that still then P sell the company that's a trend layer Foundation model from scratch because prescribing a large language model is pretty expensive and the inmate Tech, uh, several million dollars, and the only big Tech can afford it like The Meta like OpenAI, yeah. So based on our observation, most users use open-source pre-trained large language models like a metas or Lama two like Falcon like vicuna and then use their data set for the fight tuning, yeah, and the fine-tuning is will not, uh, will not cost a lot of money, but for the serving, it's a totally different story because the not because the model becomes very large, for example, like the

number 270b. In most of Industry case, we will not use the consultation, so we will use f316 for each parameter. So it's required 140 gigabytes of GPU memory, so we need multiple and the largest GPU for the inference. So it becomes very memory intensive. So there are a larger of the memory appendation technique, like the continuous batching, like the patch attention from the vln, like a fashion attention, yeah. And we have any scale published a lot of blogs, a lot about that. So if you are interested, you can take a look at it, yeah. So our problem is that the serving becomes much more expensive and important than before, right? The ml industry ml apps industry move from the training to the serving, yeah. And for the Aon with qbrew can match the end to end Ln life cycle on kubernetes, and we think that the cube and array are the best solution for large language model and kubernetes, especially for large language model serving cost. The first is that I should introduce.

that we have the auto-scaling because the online serving the workload is not it's hard to predict so we require the auto-scaling to add our data our computer resource based on the dynamic load to step the cost. The second is the heterogeneous because currently it's not it's not that you have money so you can get a GPU a lot of times that you have money but you cannot get GPU especially for some high-end GPU like a100 or h100 so supportive support the heterogeneous computer resource are very important like different types of GPU like GPUs or CPUs and then we will demo the end-to-end a large language model lifecycle with QB. Yeah this is our architecture of our demo the first is that we will do their processing with the recursor Jupiter notebook and the auto-scaling and then after we finish the data processing we will create a region for the phytuning because the workload the resource Computing resource requirement for the fine-tuning is is able to predict so we decided to use rate drop to several course.

and then after the fire tuning, it will upload the model checkpoint to the account storage. And then step three, we can use the ray service to serve our fine to large language model. And the first, therefore, we will build an LM application with the lension and the gradual to showcase the application based on our endpoints served by Race Service. Yeah, for the prototyping side, uh, in this demo, we will not cover this part, but, uh, but this is a recommendation architecture. Uh, the first is that we typically recommend user to run a Jupiter notebook as a head pass cycle container and land, uh, user. We also, in, uh, recommend user to enable the auto scaler, and the auto scaler will inject a cycle container into the headspot. And for the, and the second, for a second, we will fine-tuning with the ray job, and in this demo, we find you a llama27b on the math data set. And in the demo, we use a raychen and a disk bit. And then we have the 16 Nvidia A10 GPUs and a 24 gigabyte RAM for per GPU.

and we only find you for one Epoch, and it takes 22 minutes. Yeah, analysis our demo video. The first is that we install a qbre operator, yeah, and the ones for it should be ready, right? And then creates a great job for fine-tuning, yeah, and check the custom resource, yeah, and we watch it apart, and it one has node and the 16 GPU working nodes, and then we watch the monitor, the fine-tuning job progress, and then after 20 minutes, right, it's the state, and you can see the checkpoint here, yeah, and then because this is for demo purpose, we didn't configure the auto-detecting Direction, so after this is by ourselves, and then after we finish the phytonin, we can serve the, we can serve the fighting model with Race Service, and the ray Ln, and the real M is a previously named every it is a tool that can help you to serve your large language model in Array, and it provides open AI-compatible API, so it's pretty easy to use the open air Python package to use the, to use the.

endpoint Leicester by Ray service and this is the demo. What should you do last time oh here we go yeah. We have already have an operator that in our cluster so we create a red service yeah and then we watch our part. And in the raycaster it will have one head no and the one GPU work node. And we will serve law serve a large language model on the GPU node. And then it has a it is ready so we just try to pull forwarding and then and then use the curl request to test it yeah and we got a response cool and then uh. The last step is that we have a endpoint let's start by our array service with our fine tune llama27b model. So the last step is that I want to build an LM application with Lane Chen and the gradual and here we use graduation for to build a web UI and use the engine to handle the i o between the race service endpoints and our users inputs and output right. And because this one is a RFI 2 model is fine too with the math data set.

so we will use a math question for the demo, and the lantern has a it will use a prompt canvas. That's as we've shown in the slide, right? This is how we find to our model, so that's why we use this prompt template, yeah. And this is a math problem that we will use. It is the same from our math data set, and the answer is five. It's a pretty simple math question, yeah. And the list is the very short video, yeah, because I have already put forwarding that to my Dev box. So I just run the gradual server that's on my local box, and then you can see this UI, and we use the question that's in the last slide, and the summit should our endpoints, yeah. And the result from our model is not correct because we just find you for one Epoch, but the response is that it has already followed our data says format, so it is better than the best model, no, yeah. And uh, and uh, actually introduced the current status of the qbre, and I introd, I talked about the LM with qbre.

So currently, I want to recapital 2003 for qbre community. Uh, from the last restaurant until today, we have four major release from 230 companies to six more than 600 comments, and we have more than 90 contributors. Uh, like Dimitri from curse, and uh, joshing Wilson, and Chinese from bidance, and RD from Microsoft, and like each time from the, and several folks from the red hat and IBN, right? We have a very, very good community, and thanks for the contribution. And then we have a lot of users blocked like the, like the insta cards, uh, sansara, Spotify, and the draw Dash, and the Google, and uh, we have a massive growth in the number of qbre clusters. We have a 9x growth in the last six months, and the two acts in the August, and uh, in our expectation, we think that this, uh, this month will have another 2x in September. And this is our user's feedback, uh, from the Antics, right? Because I am not sure a lot is this a legal for me to use their.

logo. So I decided to use the Psyduck, which is my favorite Pokemon. Yeah, and uh, Niantic's with Cuberay, and they used to for less game processing, and it's definitely caused by 60 percent, right, which is very fantastic, right? So we are very happy to announce that uh, Cuberay is generally available, right? I think you guys know that in the ion's Keynote, right? And then uh, the currency, the Cuberay 1.0, the first race candy is available, so feel free to try it on. This really extended will be very different from the previous one. For the previous one, we will just iterate with user. For this case, for one to two weeks, but this one, we will iterate maybe for six weeks. So it is a good time if you have any bug, right? It will be fixed very fast, so it's a good time to open some issue, yeah, and thanks for you guys to attending our talk. Thanks.

KubeRay: A Ray cluster management solution on Kubernetes

Welcome to the Anyscale YouTube channel! Join us as we explore Ray, an open-source framework designed to accelerate the development and deployment of machine learning applications. From beginner tutorials to advanced insights, our videos cater to all skill levels. Stay up-to-date with the latest advancements and hear from experts through interviews, panel discussions, and keynotes.

Anyscale

Kubray를 소개합니다: 레이 애플리케이션을 위한 쿠버네티스 오퍼레이터, Kubray 소개

Kubray는 원활한 통합에 중점을 두고 데이터 및 머신 러닝 과학자들이 작업에 집중할 수 있도록 Kubernetes에서 Ray 클러스터와 관련 애플리케이션의 전체 수명 주기를 관리하는 Kubernetes 운영자입니다.

장기 실행 클러스터를 위한 레이 클러스터 사용자 정의 리소스, 애드혹 워크로드를 위한 레이 작업 사용자 정의 리소스, 모델 서비스를 위한 레이 서비스 사용자 정의 리소스 등 다양한 워크로드를 위한 세 가지 핵심 구성 요소를 제공합니다.

Kubray의 가장 큰 장점은 데이터/ML 과학자와 인프라 엔지니어 간의 책임 분리를 통해 Kubernetes에서 Ray를 효율적으로 관리할 수 있다는 것입니다.

사용자 지정 리소스로 레이 클러스터 관리

레이 작업 사용자 지정 리소스를 사용하면 작업 완료에 따라 레이 클러스터를 자동으로 생성 및 삭제할 수 있어 클라우드 리소스를 절약할 수 있습니다.

반면, Ray 클러스터 사용자 지정 리소스는 장기 실행 클러스터와 임시 클러스터를 모두 관리하는 데 적합하며, Ray 자동 스케일러와 원활하게 통합됩니다.

한편, Ray 서비스 사용자 지정 리소스는 가동 시간이 중요한 사용 사례에 맞춰 다운타임 없는 업그레이드와 고가용성을 보장하는 데 이상적입니다.

레이 서비스는 다운타임 없는 클러스터 변경을 제공하여 비정상적인 상태 또는 구성 변경 시 중단 없이 트래픽이 새 클러스터로 이동하도록 보장합니다.

또한 고가용성 및 GCS 내결함성을 제공하여 Ray 헤드 포드가 다운되더라도 사용자 트래픽이 계속 흐르도록 합니다.

레이 서비스의 아키텍처에는 사용자 정의 리소스 생성, 운영자의 모니터링, Kubernetes 리소스 관리가 포함되므로 레이 클러스터와 애플리케이션을 위한 솔루션으로 적합합니다.

대규모 언어 모델 서비스 비용의 중요성 증가

ML 업계에서 대규모 언어 모델 제공에 드는 비용이 크게 증가하고 있습니다.

학습에서 서빙으로 전환하면서 GPU 메모리를 많이 사용하게 되어 고가의 인프라와 메모리 최적화 기술이 필요해졌습니다.

이를 해결하기 위해서는 자동 확장 및 이기종 컴퓨터 리소스가 중요하며, Kubernetes는 대규모 언어 모델 제공 비용을 관리하는 데 가장 적합한 솔루션으로 간주됩니다.

Ray Service를 사용한 대규모 언어 모델 구축 및 배포

이 과정에는 작업에 필요한 특정 GPU 및 RAM 요구 사항에 따라 Ray를 사용하여 LLM 모델을 미세 조정하는 작업이 포함됩니다.

다음 단계는 개방형 AI 호환 API를 제공하는 Ray Service를 사용하여 미세 조정된 모델을 서비스하는 것입니다.

마지막으로, 이 계획에는 웹 UI를 만들고 레이 서비스 엔드포인트로 입출력을 관리하기 위해 레인 첸과 Gradual을 사용하여 LM 애플리케이션을 구축하는 것이 포함됩니다.

QBRE 커뮤니티는 2003년 이후 네 번의 주요 릴리스와 90명 이상의 기여자가 작성한 600개 이상의 댓글을 통해 상당한 성장을 경험했습니다.

지난 6개월 동안 QBRE 클러스터의 수는 9배 증가했으며, 다음 달에는 2배 증가할 것으로 예상됩니다.

Niantic을 비롯한 사용자 피드백은 게임 처리 속도가 60% 개선되는 등 매우 긍정적이었습니다.

이제 더 빠른 버그 수정을 위해 6주간의 반복 주기를 연장한 QBRE 1.