Choosing Graphics Cards for Machine Learning Training

Vitaly emphasizes that choosing the right graphics card is crucial for optimizing neural network training speed.

### Choosing the Right Graphics Card for Machine Learning and Neural Network Training
* 👋 Vitaly Kuliyev introduces the topic of selecting a graphics card for machine learning, focusing on **training neural networks**, alongside his co-host Ruslan.
* ⚙ Key parameters for training large language models include the **number of numerical operations per second (FLOPS)**, which directly impacts **GPU core speed and training efficiency**.
* 💻 The transfer of data from video memory to processing cores is crucial, as **bottlenecks in bandwidth and processor speed** can limit **learning and inference speeds**.
* 🖥 Vitaly shares his experience using three NVIDIA 3090 graphics cards in his server and highlights the advantages of **renting cloud servers for flexibility and cost-effectiveness**.
* 📊 He emphasizes the importance of evaluating graphics cards based on **price-to-performance ratios**, suggesting that gaming cards may **outperform server options** for specific tasks.
* 💾 Memory capacity is vital for fitting model weights into video memory, especially for large models like **Llama 70B**, which requires **significant memory** depending on the precision format used.
* 🔄 The PCI Express bus and its lanes significantly influence **data transfer speed**, which can be a **limiting factor**, particularly when **multiple graphics cards** are used.
* 🗂 Lastly, disk speed can affect training efficiency, particularly when **loading large datasets**, and modern PCI Express drives can mitigate these limitations compared to **older SATA interfaces**.

Hello everyone! My name is Vitaly Kuliyev, and today we will talk about choosing a graphics card for machine learning, specifically for training neural networks. In this episode, I will be joined by Ruslan. 

Hello everyone! Ruslan will discuss the training, and I will explain which parameters are important for training large language models. One parameter is the number of numerical operations per second (FLOPS). For example, with the NVIDIA H1, it is possible to train neural networks quickly because this model has a high FLOPS rating. However, this rating directly influences the speed of the GPU cores. 

Additionally, data needs to be transferred from the video memory to the processing cores, which also takes time. By the way, Vitaly, what can you say about the influence of GPU cores on training speed? The speed of calculations depends directly on this parameter. However, for these calculations to be performed, the data must be transferred to the graphics card chip, and there can be several bottlenecks in this process.

the bandwidth of video memory, the bandwidth of the P, the speed of the processor. If this happens, then the learning speed or the speed of inference will be limited by this parameter, the number that characterizes the performance of the video chip. It is also worth mentioning that this is tied to the bit depth of numbers, such as float 32, float 16, and others. For large language models, you usually use float 16 or quantization. Therefore, this parameter should be divided by bit depth and the performance should be evaluated for a specific bit depth. 

What graphics cards did you use for your training? Well, I have my own server with three 3090 graphics cards. I’m planning to buy more soon, and I’ll be working on training them. Before building my server, I used cloud servers because it’s convenient to rent graphics cards by the hour. I have a large video on my channel comparing the main graphics cards available in the cloud, and I conducted these tests for...

To select the right graphics cards for your server, you can see their costs. There are both gaming graphics cards like the 3090 and 490, as well as server-level options like the H1 and A1. After conducting tests in the cloud using these graphics cards, I concluded that gaming-level graphics cards offer a better price-to-performance ratio for my tasks. If your goal is to train or run neural networks, I also recommend trying different graphics card options to compare training speed or inference speed on the required neural network with specific graphics cards and their quantities. After that, you can draw conclusions and choose a specific setup to either purchase or rent for an extended period. This will directly affect both training speed and inference speed. Additionally, if you want to estimate approximately how many GPU hours you need to train your model, knowing the size of your dataset and the number of parameters in the model you plan to use will be helpful.

Where in the model card do I indicate the training graph on a specific graphics processor? In my videos, I show which graphics processor I used on the platform, and in the model card, you can see the training graph, the number of training epochs, and the time spent. If you rent a server for a long time, like a month or two, there are significant discounts, which can help save on computing costs. Finally, the amount of memory is probably the most obvious parameter of any graphics card. Why is it important, especially for training? As I mentioned, training requires fitting the model weights in Half Precision (FP16) into video memory. It is crucial for the entire model to fit in video memory. There are some frameworks, such as Lama CPPM Studio, that allow keeping part of the weights in RAM, but this greatly affects the speed of the neural network's performance. Therefore, video memory must be sufficient for the entire neural network and all weights to fit in memory, leaving additional space for...

Контекст размера или значения K в кешировании зависит от того, сколько запросов может обрабатываться параллельно. Давайте я приведу некоторые цифры. Лама 70B, как уже упомянул Руслан, имеет 70 миллиардов весов. Если запускать в полной точности, то есть FP16, требуется примерно 140 ГБ видеопамяти. Но если сравнивать с большинством видеокарт, таких как H100 или A1, у них по 80 ГБ, или же с игровыми решениями, у которых максимум 24 ГБ, как у 3090 и 4090, то становится понятно, что на одной карте в такой точности запустить нейросеть не получится. Поэтому нейросети запускают либо на нескольких картах, либо используют квантизацию. При квантизации Q8, то есть 8 бит на один вес, нейросеть будет занимать 70 ГБ. Я у себя запускаю нейросеть с AWQ квантизацией, это квантизация 4 бита, и с AWQ квантизацией нейросеть занимает примерно 38 ГБ видеопамяти. Таким образом, Лама 70B с квантизацией в 4 бита может быть запущена уже на двух видеокартах 3090.

or 4090, and there will be 10 GB of context size available from this free memory, which also depends on the speed of operation. This is because, as a rule, neural networks are used by a large number of users, and the chip usually does not develop further. This is a continuation of the idea of a regular batch, only applied to the information of large language models for predicting the next token. For example, we have 10 requests, each with its own length; they are combined, and in one batch, the next token is predicted for several requests at once. In order to implement such a scheme, it is necessary for all these requests and contexts to be located within the video memory. According to the benchmarks I conducted, for example, graphics cards 3090 and 4090 are already hitting the memory limit. If there are two graphics cards, then 10 GB is already insufficient to hold all the requests that one wants to process. As a result, it turns out that the chip is underutilized. Another important parameter for training speed is to ensure that these batches are constantly transferred from...

Видеопамять и процессор работают максимально быстро, но это зависит от пропускной способности и производительности памяти видеокарты. Этот параметр становится критически важным при небольшом количестве запросов, что часто актуально для одиночных запросов. Например, когда пользователь запускает локально небольшую модель, и ему генерируется ответ. В подобном сценарии использования акцент делается на скорость видеопамяти, потому что все веса передаются, а запросов у нас всего один. Данных для вычислений относительно мало, и передача из процессора с диска практически отсутствует. Поэтому чип быстрый, но для загрузки весов требуется время. А чтобы предсказать следующий токен в единичном запросе, нужно передать все веса на видеокарту. Именно в этом месте возникает ограничение скорости, и квантизация сильно помогает в подобном сценарии. Если, например, у вас была квантизация 8 бит, а стала 4 бита, то весов становится в два раза меньше.

It is necessary to transfer data to the graphics card from the video memory, which can lead to approximately double the acceleration. The next limiting parameter is the PCI Express bus. Currently, five generations of PCI Express have been released. For example, you might see PCI Express 4 or 5, which refers to the generation of PCI Express. This standard has been developed in such a way that each subsequent generation doubles the bandwidth of the PCI Express bus. There are also different numbers of lanes that can be connected to the graphics card: x16, x8, x4, x2, and x1. The more lanes that are connected to the graphics card, the faster the bandwidth will be, which is directly proportional. For instance, if we take a PCI Express 3.0 connection with x16, the bandwidth will approximately correspond to PCI Express 4.0. In the context of tasks, the bandwidth has secondary importance. According to my tests, this is crucial during the initial query when all input tokens need to be processed. However, during token generation, the bandwidth can have a minor significance, especially when using multiple graphics cards.

Во многих случаях вся экспрессия является ограничивающим фактором, потому что все градиенты с одной видеокарты нужно передать на другую. В частности, стоит упомянуть технологии NVIDIA, такие как NV Switch и NVLink, которые по-разному называются. Данная технология поддерживается, как правило, серверами видеокартами, в то время как в игровых видеокартах поддержка NVLink закончилась на 3000-м поколении. То есть 3090 имеет NVLink, а 4090 уже не имеет этого подключения. Поэтому, если нужно тренировать большие модели, это как правило делают на серверных видеокартах H100 и A100, которые подключены к высокоскоростной шине NV. Потому что на данный момент Pess медленнее, чем проприетарное решение от NVIDIA. 

Ещё нужно рассказать о процессоре. Именно центральный процессор в ряде случаев может являться ограничивающим фактором обучения. На процессоре, как правило, готовятся данные, и после этого они передаются на видеокарту. В задачах инса процессоры имеют второстепенную роль. Туда подаются текстовые данные.

Language models transmit text to the graphics card, and there, the situation becomes more complex with images because images are usually processed by the CPU. If we need to perform operations on images, the image is loaded either from disk or over the network. Then, the central processor decompresses it, usually in formats like GP, PNG, or other formats. This compressed image format is transformed into a matrix of the required dimensions, and this data is then sent to the graphics card. At this stage, the processor can be a limiting factor. When it comes to training with images, augmentation is typically used. Augmentation is a process of making small changes to the data so that the model does not memorize the original image. For example, if we have a dataset of cat images that is relatively small, we need to ensure that the model doesn't just memorize these images. Therefore, we apply augmentation techniques, such as slight changes in brightness or small shifts of the image to the left or right.

scaling up and down, cropping, changing gamma, and a number of other parameters. These are small changes that the human eye can perceive; for example, the same cat might appear slightly darker, or its position might have shifted a bit. However, for a neural network, this represents a completely different image because the matrix has fundamentally changed, and it learns to draw conclusions based on the augmented image. 

At one of my workplaces, we had such a scenario. There was a server with two graphics cards and an eight-core processor. We were running two training processes, specifically focused on image augmentation, and the emphasis was on the processor. When we launched training on one graphics card, the processor was loaded to about six cores. When we simultaneously launched a different variant on the second graphics card, the processor was fully utilized with all eight cores, and the training of both networks was happening more slowly. Therefore, it is essential to consider parameters like these.

Performance across all cores. Next, I want to talk about the support for new instructions such as FP8 and FP4 in NVIDIA graphics cards, specifically the H100 and newer H200 models. Support for new instructions like FP8 has been introduced primarily for neural networks. As I mentioned, neural networks are usually run externally in a lower dimensionality. For example, the H100 supports operations with the FP8 format. If you load a network in the FP8 format, which is 8 bits, the H100 graphics card can natively support these instructions. However, if we consider the previous generation of graphics cards, such as the A100, 3090, or earlier versions, there is no support for this instruction, and it is not natively supported. For instance, when quantizing models like AWQ or GG in Q8, we first take the weights, convert them to the FP16 format, and then all operations occur in the FP16 format on the graphics chip. The result is then returned, meaning that the previous generation operates in this manner.

Сценарии будут медленнее, чем новейшие поколения, такие как H1 и H200 видеокарт, и на них есть нативная поддержка этих инструкций. Это стоит учитывать для тренировки нейросетей. Задача тренировки нейросетей в FP8 редко применяется, чаще всего используется FP16 для тренировки модели. Но я думаю, что с ростом моделей и увеличением количества видеокарт, которые поддерживают нативно FP8 и более низкие размерности, такой формат будет более популярен. Университеты тоже будут тренироваться в FP8 и FP4 форматах, и видеокарты нового поколения будут выгодно отличаться от предыдущих поколений. Я думаю, помимо поддержки форматов на уровне железа, еще значение имеет наличие библиотек, которые поддерживают способ обучения. Наверное, чтобы облегчить вход для инженеров, NVIDIA выпустила свой фреймворк Transform Engine, с помощью которого можно обучать в FP8 на H100. Но, конечно, большинство инженеров сейчас сосредоточено на стеке Henf Transformers, где люди привыкли обучать.

The last question is about the disk and how it affects training. Neural networks typically store model checkpoints and data on the disk, and in rare cases, the disk can become a bottleneck. Let's consider a scenario where this might be important. For instance, if we are training a small neural network, perhaps for quick inference on a mobile phone or embedded devices like Jetson Nano or Jetson Orion, the size of the network is small, but we want to train it on a large amount of data. In this case, we have a large dataset of images that need to be loaded from the disk, augmented, and then sent to the graphics card. If we have a sufficiently small neural network and a fast graphics card, we may encounter limitations due to the speed of the disk. This is particularly relevant for older disks with SATA interfaces.

500 MB per second is the reading speed, and just in this interface, there will be more modern drives with PCI Express connectivity. These drives are either inserted directly into the PCI Express slot or connected via interface 2. In that case, this issue becomes less relevant. If we consider, for example, the 70B, the speed of the drive will be determined by how quickly the model starts and loads into the video memory. For this process, it is necessary to read all the weights from the drive. I would like to remind you that even when considering quantization at 140 GB, these 140 GB need to be read and transferred to the graphics card's memory, and it is here that the drive will be a limiting factor. As for ISA, it is usually not that important to wait for the start to finish; what matters more is how it operates. Therefore, we can even use older drives. Today's release was in a Colab format. I recommend Ruslan's channel, which will be linked in the description of this video. Go check it out; he talks about training and also offers a different perspective on it.

I would also like to remind you that the sponsor of this video is Cloud, a company that provides servers with all these graphics cards. You can take advantage of their services; the link will also be in the description of this video. This was Vitaly Kuliev. Goodbye, everyone!

Топ советов по выбору видеокарты для машинного обучения и инференса нейросетей(LLM)?

Виталий Кулиев

Choosing a Graphics Card for Machine Learning

Vitaly and Ruslan discuss the important parameters for training neural networks, with a focus on FLOPS and GPU core speed.

They highlight the significance of data transfer and potential bottlenecks in the training process.

Vitaly shares his experience with using 3090 graphics cards and recommends comparing different options for the best price-to-performance ratio in training neural networks.

The Importance of Graphics Processor and Memory in Model Training

The training graph, number of epochs, and time spent are indicated in the model card, along with the graphics processor used.

Sufficient video memory is crucial for fitting model weights in Half Precision (FP16), with larger models like Llama 70B requiring 140 GB.

Q8 and AWQ quantization can reduce memory requirements, allowing the model to run on fewer GPUs while also providing context size for parallel request processing.

Влияние Видеопамяти и Процессора на Производительность

Видеопамять и производительность графической карты критически важны при небольшом количестве запросов, как в случае с локальным запуском модели.

Квантизация весов помогает ускорить передачу данных на видеокарту, что может привести к двойному ускорению.

PCI Express играет важную роль в пропускной способности, а технологии NVIDIA, такие как NV Switch и NVLink, используются для соединения серверных видеокарт при тренировке больших моделей.

The Complex Process of Image Processing in Neural Networks

Language models transmit text to the graphics card, but image processing is more complex, involving the CPU for operations.

Image augmentation is used during training to prevent models from memorizing original images, with techniques like brightness changes and image shifts.

The newer NVIDIA graphics cards, like the H100 and H200, support FP8 and FP4 instructions for neural networks, while previous generations lack this native support.

The Impact of GPU Generations and Disk Speed on Neural Network Training

Newer GPU generations, like H1 and H200, have native support for FP8 and lower precision formats, which may become more popular for neural network training.

Universities are also expected to adopt FP8 and FP4 formats, giving a competitive edge to next-gen GPUs.

The disk can become a bottleneck in training, especially when loading large datasets for small neural networks, with older SATA disks being particularly problematic.

The Impact of Drive Speed on Model Loading

The reading speed of 500 MB per second is crucial for modern drives with PCI Express connectivity.

The drive's speed determines how quickly the model loads into video memory, making it a limiting factor for transferring weights.

The video also promotes Ruslan's channel and Cloud's server services for graphics cards.