머신 러닝 트레이닝을 위한 그래픽 카드 선택

비탈리는 신경망 훈련 속도를 최적화하려면 올바른 그래픽 카드를 선택하는 것이 중요하다고 강조합니다.

### 머신 러닝 및 신경망 훈련에 적합한 그래픽 카드 선택하기
* 👋 비탈리 쿨리예프가 공동 진행자인 루슬란과 함께 **신경망 훈련에 초점을 맞춘 머신러닝용 그래픽 카드 선택**에 대한 주제를 소개합니다.
* ⚙ 대규모 언어 모델 학습을 위한 주요 파라미터에는 GPU 코어 속도와 학습 효율에 직접적인 영향을 미치는 **초당 숫자 연산 수(FLOPS)**가 포함됩니다.
* 💻 대역폭과 프로세서 속도의 병목 현상으로 인해 학습 및 추론 속도가 제한될 수 있으므로 **비디오 메모리에서 처리 코어로 데이터를 전송하는 것**은 매우 중요합니다.
* 🖥 Vitaly는 자신의 서버에서 **세 대의 NVIDIA 3090 그래픽 카드**를 사용한 경험을 공유하고 유연성과 비용 효율성을 위한 **클라우드 서버 대여의 이점**을 강조합니다.
* 📊 그는 가격 대비 성능 비율을 기준으로 그래픽 카드를 평가하는 것이 중요하다고 강조하며, **특정 작업에서는 게이밍 카드가 서버 옵션보다 성능이 우수**할 수 있다고 제안합니다.
* 💾 메모리 용량은 모델 가중치를 비디오 메모리에 맞추는 데 필수적이며, 특히 사용되는 정밀 포맷에 따라 상당한 메모리가 필요한 **Llama 70B와 같은 대형 모델**의 경우 더욱 그렇습니다.
* 🔄 PCI Express 버스와 그 레인은 데이터 전송 속도에 큰 영향을 미치며, 특히 **여러 개의 그래픽 카드를 사용하는 경우 데이터 전송 속도를 제한하는 요인**이 될 수 있습니다.
* 🗂 마지막으로, 디스크 속도는 특히 대용량 데이터 세트를 로드할 때 교육 효율성에 영향을 미칠 수 있으며, **최신 PCI Express 드라이브는 구형 SATA 인터페이스에 비해 이러한 제한을 완화**할 수 있습니다.

Hello everyone! My name is Vitaly Kuliyev, and today we will talk about choosing a graphics card for machine learning, specifically for training neural networks. In this episode, I will be joined by Ruslan. 

Hello everyone! Ruslan will discuss the training, and I will explain which parameters are important for training large language models. One parameter is the number of numerical operations per second (FLOPS). For example, with the NVIDIA H1, it is possible to train neural networks quickly because this model has a high FLOPS rating. However, this rating directly influences the speed of the GPU cores. 

Additionally, data needs to be transferred from the video memory to the processing cores, which also takes time. By the way, Vitaly, what can you say about the influence of GPU cores on training speed? The speed of calculations depends directly on this parameter. However, for these calculations to be performed, the data must be transferred to the graphics card chip, and there can be several bottlenecks in this process.

the bandwidth of video memory, the bandwidth of the P, the speed of the processor. If this happens, then the learning speed or the speed of inference will be limited by this parameter, the number that characterizes the performance of the video chip. It is also worth mentioning that this is tied to the bit depth of numbers, such as float 32, float 16, and others. For large language models, you usually use float 16 or quantization. Therefore, this parameter should be divided by bit depth and the performance should be evaluated for a specific bit depth. 

What graphics cards did you use for your training? Well, I have my own server with three 3090 graphics cards. I’m planning to buy more soon, and I’ll be working on training them. Before building my server, I used cloud servers because it’s convenient to rent graphics cards by the hour. I have a large video on my channel comparing the main graphics cards available in the cloud, and I conducted these tests for...

To select the right graphics cards for your server, you can see their costs. There are both gaming graphics cards like the 3090 and 490, as well as server-level options like the H1 and A1. After conducting tests in the cloud using these graphics cards, I concluded that gaming-level graphics cards offer a better price-to-performance ratio for my tasks. If your goal is to train or run neural networks, I also recommend trying different graphics card options to compare training speed or inference speed on the required neural network with specific graphics cards and their quantities. After that, you can draw conclusions and choose a specific setup to either purchase or rent for an extended period. This will directly affect both training speed and inference speed. Additionally, if you want to estimate approximately how many GPU hours you need to train your model, knowing the size of your dataset and the number of parameters in the model you plan to use will be helpful.

Where in the model card do I indicate the training graph on a specific graphics processor? In my videos, I show which graphics processor I used on the platform, and in the model card, you can see the training graph, the number of training epochs, and the time spent. If you rent a server for a long time, like a month or two, there are significant discounts, which can help save on computing costs. Finally, the amount of memory is probably the most obvious parameter of any graphics card. Why is it important, especially for training? As I mentioned, training requires fitting the model weights in Half Precision (FP16) into video memory. It is crucial for the entire model to fit in video memory. There are some frameworks, such as Lama CPPM Studio, that allow keeping part of the weights in RAM, but this greatly affects the speed of the neural network's performance. Therefore, video memory must be sufficient for the entire neural network and all weights to fit in memory, leaving additional space for...

Контекст размера или значения K в кешировании зависит от того, сколько запросов может обрабатываться параллельно. Давайте я приведу некоторые цифры. Лама 70B, как уже упомянул Руслан, имеет 70 миллиардов весов. Если запускать в полной точности, то есть FP16, требуется примерно 140 ГБ видеопамяти. Но если сравнивать с большинством видеокарт, таких как H100 или A1, у них по 80 ГБ, или же с игровыми решениями, у которых максимум 24 ГБ, как у 3090 и 4090, то становится понятно, что на одной карте в такой точности запустить нейросеть не получится. Поэтому нейросети запускают либо на нескольких картах, либо используют квантизацию. При квантизации Q8, то есть 8 бит на один вес, нейросеть будет занимать 70 ГБ. Я у себя запускаю нейросеть с AWQ квантизацией, это квантизация 4 бита, и с AWQ квантизацией нейросеть занимает примерно 38 ГБ видеопамяти. Таким образом, Лама 70B с квантизацией в 4 бита может быть запущена уже на двух видеокартах 3090.

or 4090, and there will be 10 GB of context size available from this free memory, which also depends on the speed of operation. This is because, as a rule, neural networks are used by a large number of users, and the chip usually does not develop further. This is a continuation of the idea of a regular batch, only applied to the information of large language models for predicting the next token. For example, we have 10 requests, each with its own length; they are combined, and in one batch, the next token is predicted for several requests at once. In order to implement such a scheme, it is necessary for all these requests and contexts to be located within the video memory. According to the benchmarks I conducted, for example, graphics cards 3090 and 4090 are already hitting the memory limit. If there are two graphics cards, then 10 GB is already insufficient to hold all the requests that one wants to process. As a result, it turns out that the chip is underutilized. Another important parameter for training speed is to ensure that these batches are constantly transferred from...

Видеопамять и процессор работают максимально быстро, но это зависит от пропускной способности и производительности памяти видеокарты. Этот параметр становится критически важным при небольшом количестве запросов, что часто актуально для одиночных запросов. Например, когда пользователь запускает локально небольшую модель, и ему генерируется ответ. В подобном сценарии использования акцент делается на скорость видеопамяти, потому что все веса передаются, а запросов у нас всего один. Данных для вычислений относительно мало, и передача из процессора с диска практически отсутствует. Поэтому чип быстрый, но для загрузки весов требуется время. А чтобы предсказать следующий токен в единичном запросе, нужно передать все веса на видеокарту. Именно в этом месте возникает ограничение скорости, и квантизация сильно помогает в подобном сценарии. Если, например, у вас была квантизация 8 бит, а стала 4 бита, то весов становится в два раза меньше.

It is necessary to transfer data to the graphics card from the video memory, which can lead to approximately double the acceleration. The next limiting parameter is the PCI Express bus. Currently, five generations of PCI Express have been released. For example, you might see PCI Express 4 or 5, which refers to the generation of PCI Express. This standard has been developed in such a way that each subsequent generation doubles the bandwidth of the PCI Express bus. There are also different numbers of lanes that can be connected to the graphics card: x16, x8, x4, x2, and x1. The more lanes that are connected to the graphics card, the faster the bandwidth will be, which is directly proportional. For instance, if we take a PCI Express 3.0 connection with x16, the bandwidth will approximately correspond to PCI Express 4.0. In the context of tasks, the bandwidth has secondary importance. According to my tests, this is crucial during the initial query when all input tokens need to be processed. However, during token generation, the bandwidth can have a minor significance, especially when using multiple graphics cards.

Во многих случаях вся экспрессия является ограничивающим фактором, потому что все градиенты с одной видеокарты нужно передать на другую. В частности, стоит упомянуть технологии NVIDIA, такие как NV Switch и NVLink, которые по-разному называются. Данная технология поддерживается, как правило, серверами видеокартами, в то время как в игровых видеокартах поддержка NVLink закончилась на 3000-м поколении. То есть 3090 имеет NVLink, а 4090 уже не имеет этого подключения. Поэтому, если нужно тренировать большие модели, это как правило делают на серверных видеокартах H100 и A100, которые подключены к высокоскоростной шине NV. Потому что на данный момент Pess медленнее, чем проприетарное решение от NVIDIA. 

Ещё нужно рассказать о процессоре. Именно центральный процессор в ряде случаев может являться ограничивающим фактором обучения. На процессоре, как правило, готовятся данные, и после этого они передаются на видеокарту. В задачах инса процессоры имеют второстепенную роль. Туда подаются текстовые данные.

Language models transmit text to the graphics card, and there, the situation becomes more complex with images because images are usually processed by the CPU. If we need to perform operations on images, the image is loaded either from disk or over the network. Then, the central processor decompresses it, usually in formats like GP, PNG, or other formats. This compressed image format is transformed into a matrix of the required dimensions, and this data is then sent to the graphics card. At this stage, the processor can be a limiting factor. When it comes to training with images, augmentation is typically used. Augmentation is a process of making small changes to the data so that the model does not memorize the original image. For example, if we have a dataset of cat images that is relatively small, we need to ensure that the model doesn't just memorize these images. Therefore, we apply augmentation techniques, such as slight changes in brightness or small shifts of the image to the left or right.

scaling up and down, cropping, changing gamma, and a number of other parameters. These are small changes that the human eye can perceive; for example, the same cat might appear slightly darker, or its position might have shifted a bit. However, for a neural network, this represents a completely different image because the matrix has fundamentally changed, and it learns to draw conclusions based on the augmented image. 

At one of my workplaces, we had such a scenario. There was a server with two graphics cards and an eight-core processor. We were running two training processes, specifically focused on image augmentation, and the emphasis was on the processor. When we launched training on one graphics card, the processor was loaded to about six cores. When we simultaneously launched a different variant on the second graphics card, the processor was fully utilized with all eight cores, and the training of both networks was happening more slowly. Therefore, it is essential to consider parameters like these.

Performance across all cores. Next, I want to talk about the support for new instructions such as FP8 and FP4 in NVIDIA graphics cards, specifically the H100 and newer H200 models. Support for new instructions like FP8 has been introduced primarily for neural networks. As I mentioned, neural networks are usually run externally in a lower dimensionality. For example, the H100 supports operations with the FP8 format. If you load a network in the FP8 format, which is 8 bits, the H100 graphics card can natively support these instructions. However, if we consider the previous generation of graphics cards, such as the A100, 3090, or earlier versions, there is no support for this instruction, and it is not natively supported. For instance, when quantizing models like AWQ or GG in Q8, we first take the weights, convert them to the FP16 format, and then all operations occur in the FP16 format on the graphics chip. The result is then returned, meaning that the previous generation operates in this manner.

Сценарии будут медленнее, чем новейшие поколения, такие как H1 и H200 видеокарт, и на них есть нативная поддержка этих инструкций. Это стоит учитывать для тренировки нейросетей. Задача тренировки нейросетей в FP8 редко применяется, чаще всего используется FP16 для тренировки модели. Но я думаю, что с ростом моделей и увеличением количества видеокарт, которые поддерживают нативно FP8 и более низкие размерности, такой формат будет более популярен. Университеты тоже будут тренироваться в FP8 и FP4 форматах, и видеокарты нового поколения будут выгодно отличаться от предыдущих поколений. Я думаю, помимо поддержки форматов на уровне железа, еще значение имеет наличие библиотек, которые поддерживают способ обучения. Наверное, чтобы облегчить вход для инженеров, NVIDIA выпустила свой фреймворк Transform Engine, с помощью которого можно обучать в FP8 на H100. Но, конечно, большинство инженеров сейчас сосредоточено на стеке Henf Transformers, где люди привыкли обучать.

The last question is about the disk and how it affects training. Neural networks typically store model checkpoints and data on the disk, and in rare cases, the disk can become a bottleneck. Let's consider a scenario where this might be important. For instance, if we are training a small neural network, perhaps for quick inference on a mobile phone or embedded devices like Jetson Nano or Jetson Orion, the size of the network is small, but we want to train it on a large amount of data. In this case, we have a large dataset of images that need to be loaded from the disk, augmented, and then sent to the graphics card. If we have a sufficiently small neural network and a fast graphics card, we may encounter limitations due to the speed of the disk. This is particularly relevant for older disks with SATA interfaces.

500 MB per second is the reading speed, and just in this interface, there will be more modern drives with PCI Express connectivity. These drives are either inserted directly into the PCI Express slot or connected via interface 2. In that case, this issue becomes less relevant. If we consider, for example, the 70B, the speed of the drive will be determined by how quickly the model starts and loads into the video memory. For this process, it is necessary to read all the weights from the drive. I would like to remind you that even when considering quantization at 140 GB, these 140 GB need to be read and transferred to the graphics card's memory, and it is here that the drive will be a limiting factor. As for ISA, it is usually not that important to wait for the start to finish; what matters more is how it operates. Therefore, we can even use older drives. Today's release was in a Colab format. I recommend Ruslan's channel, which will be linked in the description of this video. Go check it out; he talks about training and also offers a different perspective on it.

I would also like to remind you that the sponsor of this video is Cloud, a company that provides servers with all these graphics cards. You can take advantage of their services; the link will also be in the description of this video. This was Vitaly Kuliev. Goodbye, everyone!

Топ советов по выбору видеокарты для машинного обучения и инференса нейросетей(LLM)?

Виталий Кулиев

머신 러닝을 위한 그래픽 카드 선택

비탈리와 루슬란이 신경망 훈련의 중요한 매개변수에 대해 설명하며, FLOPS와 GPU 코어 속도에 중점을 두고 설명합니다.

이들은 데이터 전송의 중요성과 훈련 과정에서 발생할 수 있는 병목 현상에 대해 강조합니다.

비탈리는 3090 그래픽 카드 사용 경험을 공유하고 신경망 훈련에서 최고의 가성비를 위해 다양한 옵션을 비교해 볼 것을 권장합니다.

모델 훈련에서 그래픽 프로세서와 메모리의 중요성

학습 그래프, 에포크 수, 소요 시간은 사용된 그래픽 프로세서와 함께 모델 카드에 표시됩니다.

반정밀도(FP16)로 모델 가중치를 맞추려면 충분한 비디오 메모리가 중요하며, Llama 70B와 같은 대형 모델의 경우 140GB가 필요합니다.

Q8 및 AWQ 양자화는 메모리 요구 사항을 줄여 더 적은 수의 GPU에서 모델을 실행하는 동시에 병렬 요청 처리를 위한 컨텍스트 크기를 제공할 수 있습니다.

비디오 메모리와 프로세서가 성능에 미치는 영향

로컬 모델 실행의 경우처럼 요청 수가 적을 때는 비디오 메모리와 그래픽 카드 성능이 중요합니다.

가중치 정량화는 그래픽 카드로의 데이터 전송 속도를 높여 두 배의 가속화를 가져올 수 있습니다.

PCI Express는 대역폭에서 중요한 역할을 하며, 대규모 모델을 훈련할 때 서버 그래픽 카드를 연결하는 데 NV Switch 및 NVLink와 같은 NVIDIA 기술이 사용됩니다.

신경망에서 이미지 처리의 복잡한 과정

언어 모델은 텍스트를 그래픽 카드로 전송하지만 이미지 처리는 더 복잡하여 연산을 위해 CPU가 필요합니다.

이미지 증강은 밝기 변경 및 이미지 이동과 같은 기술을 사용하여 모델이 원본 이미지를 암기하지 못하도록 훈련 중에 사용됩니다.

H100 및 H200과 같은 최신 NVIDIA 그래픽 카드는 신경망에 대한 FP8 및 FP4 명령어를 지원하지만 이전 세대에는 이러한 기본 지원이 부족합니다.

GPU 세대와 디스크 속도가 신경망 훈련에 미치는 영향

H1 및 H200과 같은 최신 GPU 세대는 신경망 훈련에 더 많이 사용될 수 있는 FP8 및 저정밀 포맷을 기본적으로 지원합니다.

대학에서도 FP8 및 FP4 형식을 채택하여 차세대 GPU에 경쟁력을 부여할 것으로 예상됩니다.

디스크는 특히 소규모 신경망을 위한 대규모 데이터 세트를 로드할 때 훈련에 병목 현상을 일으킬 수 있으며, 구형 SATA 디스크는 특히 문제가 될 수 있습니다.

드라이브 속도가 모델 로딩에 미치는 영향

PCI Express 연결이 가능한 최신 드라이브의 경우 초당 500MB의 읽기 속도가 매우 중요합니다.

드라이브의 속도는 모델이 비디오 메모리에 얼마나 빨리 로드되는지를 결정하므로 무게를 전송하는 데 제한적인 요소가 됩니다.

이 동영상은 또한 Ruslan의 채널과 Cloud의 그래픽 카드용 서버 서비스를 홍보합니다.