슬럼 이해하기: 작업 스케줄링 및 리소스 할당

컴퓨팅 클러스터에서 널리 사용되는 작업 스케줄러인 Slurm은 작업 제출 및 관리를 위해 srun, sbatch, squeue와 같은 명령을 제공합니다. 작업을 제출하고 리소스를 효율적으로 지정하는 방법을 알아보세요.

### Slurm으로 작업하기: 작업 예약 및 제출
* 📋  Slurm은 컴퓨팅 클러스터에서 작업을 제출하고 관리하기 위해 널리 사용되는 작업 스케줄러입니다.
* ⚙️  sbatch`, `srun`, `salloc`과 같은 명령은 작업을 요청하고 제출하는 데 사용되며 `scancel`은 대기열에서 작업을 취소하는 데 사용됩니다.
* 🔍  squeue`, `sinfo`, `sacct`와 같은 명령은 작업 대기열을 쿼리하고, 클러스터 정보 및 환경의 작업/사용자 세부 정보를 계산하는 데 사용됩니다.
* 🖥️  Slurm으로 작업을 제출할 때는 작업의 리소스를 정의하기 위해 CPU 수, CPU당 메모리, 시간 등의 매개변수를 지정해야 합니다.
* ⏱️  작업 대기열에서 대기 시간을 최소화하려면 필요한 리소스를 효율적으로 요청하는 것이 중요합니다.
* ✨  sbatch` 명령을 사용하면 지정된 리소스, 작업 이름, 실행할 명령으로 일괄 프로세스를 Slurm 대기열에 제출할 수 있습니다.
* 🏃‍♀️  srun` 명령은 지정된 명령을 적극적으로 실행하는 차단 호출인 반면, `salloc` 명령은 아무것도 실행하지 않고 리소스만 할당합니다.

a reasonably popular job schedule that is deployed on a number of computing clusters is slurm. Slurm is just like any other job schedule, like grid engine or others like that, that allows jobs to be submitted to it, manages the queue, and then pushes those jobs onto the compute clusters for operation. Now, there are a few commands that allow you to request jobs: `snes batch`, `srun`, `salec`. These are commands that allow you to request a job, allocate, and put a job on the job queue. `scancel`, as the name probably sounds, is the command that lets you cancel a job off of the queue. Then there's a handful of commands that allow you to look at the various states of the system. `squeue`, as it suggests, is the tool that queries the slurm queue, the job queue itself. `sinfo` queries information about the compute cluster, and `saccount` will query information about jobs or users, and other aspects of the environment. These are the three, these are the handful of commands.

if you can, if you can, you know, navigate your way around, around, you know, s batch, s cancel, and sq um, and maybe, s info, you've got pretty much everything you need to, to work through. This is there's a few other commands but these are the big ones that that allow you to interact and and work with the system. So how do you go about now submitting a job to, to the to a cluster that uses slurm, for example? Well, s batch, s run, and s alec all have parameters and there's a common set of parameters that they all use that define the jobs. So these three commands allow you to submit jobs into the queue for the slum scheduler to manage and when you're submitting a job, there are some defaults but generally the defaults aren't what you want. You want to specify additional parameters that that describe to slurm what type of job you're you're requesting with what type of resources. So some one thing that could be important to know, for example, is how many cpus per task and we're only

going to talk about a single task a job for now. So in this case, how many CPUs are you going to be requesting? That's specified with the dash lowercase c flag. So you would say something like s batch dash c, and then you would put the number in of the number of CPUs you're asking for. If you only want a single CPU per task, right per your job in this case, if there's only one task, then you're asking for c1. If you need 10 CPUs, right, because you want to run something that's going to use 10 CPUs concurrently at the same time, then you would say dash c10. You just, you ask, you're telling the Slurm how many CPUs you need for your job on a per task basis. Then, if you, you can also specify or you need to specify the memory. So how much memory per CPU are you asking for? So if you're asking for say a gigabyte of memory per CPU, then you would put in here, you know, 1g.

and this is this is pretty intelligent. You can it actually knows the suffixes so you can you know say 100 megabytes, right? I think you actually just do 100 m. I think it understands that. Or you can do something like 4g and you ask for four gigs of memory. And these are on a per cpu basis. So if you asked for dash c3, right? Three cpus, but then you specify dash dash mem per cpu and you said, you know, four gigs, then your job in total is actually going to be allocating 12 gigabytes of memory because it's this is a per cpu basis. And then the last parameter that's, you know, relatively important specify is time. How long do you need the resources of the cpu and the memory that you're asking for? How long do you need them for? That you have to know how long the job that you're going to that you're submitting is going to run. Now you might say, well, I have no idea how long this is going to run. Well, you can just try and guess some things.

and then you can go back and look to see how long it actually took, or you can underestimate it, maybe, and find that it didn't run. And then you have to, you know, jump it up a bit more. But this is something that you'll have to sort of get some intuition on to get a sense of how long it takes for a particular for a particular job to execute. Now you could say, well, why don't I just ask for as much time as possible? Well, in that case, you might be sitting around on the job queue for a very long time. It actually is in your interest to ask for as few resources as possible, both in CPUs, memory, and time, because that makes it much easier for the job scheduler to slot you in when there's an opening, right? If there's a much bigger job sitting around, then that's much harder to position into the queue or position to find resources for than a much smaller job. So if you don't want your job to be sitting around in the queue for quite some time.

it's best. It's in your best interest to trim these to just what you actually need. And time is specified. You know in units as well. So it's done with. It's done with um. It's done with some simple syntax. So you would say something like, you know, 10 colon colon 0 if you wanted 10 minutes, right? Or if you had, you know, colon colon 0 here, this would be 10 hours of time. And if you wanted like 3 days, it would be dash zero colon zero colon zero. This is three days of time being allocated. It's a pretty simple syntax. So just to round it out completely, let me show you a full sbatch example. And this, for example, is an example call that will submit a batch process to the serm queue. So in this particular example, you'll notice I'm asking for two CPUs. I'm asking for two gigs per CPU. So my job will take up four total gigabytes of space of memory. And I'm asking that for five hours.

all right because this would be seconds, minutes, and hours. I'm introducing a new parameter here, j, which I hadn't told you about. That's just the job name. I'm labeling it in the job queue so when you sq it, you can actually identify quickly what you've submitted. I'm calling this job the crunchy name. That's what I'm naming it. And then this part I hadn't shown you before, but this --wrap is an instruction to sbatch on actually the call to make, meaning what is it actually going to be submitting when it gets allocated, what command is it going to be running. And that's what this is. It's going to be running the python crunchy.py call. So you can type in anything you want here and it will execute. You can point it to a script and it can execute that. There's another, there's also other ways of writing these sbatch calls with command files, but this is a simple command line way of doing it. So this right is for sbatch if you wanted.

s run takes all these same flags, but it doesn't allow you to do the wrap command. Instead, sron is a blocking call where it actually will sit here and wait. You won't have access to the terminal until that s s1 is complete because you've specified at the end what you're running at the end of this. So there's no wrap call. It's actually all in line. And so python would appear in here, and this would wait, try to allocate resources. Once it allocates resources, it would then actively run this command, and your command line would just be waiting there until it was done, and you'd see all the output. S batch instead submits the job queue and returns control to you so that you can go about doing other things while you're waiting. And then just to be complete, s alak is all of the same parameters, only there's no wrap command here. It just lets you declare resources, and it will allocate them whenever it gets a chance, and then it will do anything else. It'll just be a job that's out there.

without anything running. And then you can jump into the compute into the system and work on it in an interactive sense if you want it or submit it manually. S Alex just allocates resources. It doesn't actually run anything. If you want to allocate resources then run something, you have to use either sbatch, which is the one that will submit things in the background, or s run, which will submit things in the foreground and actively execute them while you watch. But it could take, you know, they both, they both take the same amount of time. It's just whether or not you have control over your terminal in the meantime.

Slurm Job Scheduler Basics

Building Intuition

SLURM은 컴퓨팅 클러스터에서 작업을 제출, 관리 및 실행하기 위해 널리 사용되는 작업 스케줄러입니다.

SLURM과 상호 작용하기 위한 주요 명령으로는 작업 제출을 위한 `sbatch`, `srun`, `salloc`, 작업 취소를 위한 `scancel` 등이 있습니다.

또한 `squeue`, `sinfo`, `saccount` 등의 명령을 통해 시스템 상태 및 작업 정보를 볼 수 있습니다.

SLURM으로 작업을 제출할 때는 기본값이 적합하지 않을 수 있으므로 작업당 CPU 수와 같은 매개 변수를 지정하는 것이 중요합니다.

슬럼에서 CPU, 메모리 및 지속 시간 요청하기

"-c" 플래그를 사용하여 작업에 필요한 CPU 수를 지정합니다.

여기서 "-c1"은 CPU 1개를, "-c10"은 10개를 지정합니다.

CPU당 메모리는 "--mem-per-cpu" 플래그를 사용하여 지정합니다(예: 1기가바이트의 경우 "1g").

작업에 필요한 리소스의 지속 시간은 시간 매개변수로 표시되며, 작업이 실행되는 시간을 지정합니다.

효율적인 작업 스케줄링을 위한 팁

대기열에서 오랜 대기 시간을 피하려면 작업에 필요한 적절한 리소스 양을 예측하는 것이 중요합니다.

CPU, 메모리, 시간 측면에서 필요한 최소한의 리소스를 요청하면 작업 스케줄러가 작업을 더 쉽게 수용할 수 있습니다.

시간 할당은 10분의 경우 "10:00", 3일의 경우 "0-3:00:00"과 같은 간단한 구문을 사용하여 지정합니다.

작업 대기열에서 불필요한 지연을 피하려면 리소스 요구 사항을 신중하게 고려하는 것이 중요합니다.

sbatch로 작업 라벨링 및 제출 소개

작업 대기열의 작업 레이블 지정에 새로운 매개변수 "j"가 도입되었습니다.

"--wrap" 명령은 sbatch에서 실행할 명령을 지정하는 데 사용되며, "srun"은 지정된 명령이 완료될 때까지 기다리는 차단 호출입니다.

반면에 "sbatch"는 작업 큐를 제출하고 사용자에게 제어권을 반환하여 대기하는 동안 다른 작업을 수행할 수 있도록 합니다.

"salloc"은 자원을 선언하고 가능한 경우 랩 명령 없이 자원을 할당하여 별도의 작업으로 작동합니다.

컴퓨팅의 리소스 할당 및 실행 이해하기

컴퓨팅에서 리소스를 할당하는 작업은 sbatch 또는 srun과 같은 명령을 통해 수행됩니다.

sbatch는 백그라운드에서 작업을 제출하는 데 사용되며, srun은 포그라운드에서 작업을 제출하는 데 사용됩니다.

두 방법 모두 동일한 시간이 걸리지만 sbatch를 사용하면 실행 중에 다른 터미널을 제어할 수 있습니다.