Hello everyone! My name is Vitaly Kuliyev, and today we will talk about choosing a graphics card for machine learning, specifically for training neural networks. In this episode, I will be joined by Ruslan. 

Hello everyone! Ruslan will discuss the training, and I will explain which parameters are important for training large language models. One parameter is the number of numerical operations per second (FLOPS). For example, with the NVIDIA H1, it is possible to train neural networks quickly because this model has a high FLOPS rating. However, this rating directly influences the speed of the GPU cores. 

Additionally, data needs to be transferred from the video memory to the processing cores, which also takes time. By the way, Vitaly, what can you say about the influence of GPU cores on training speed? The speed of calculations depends directly on this parameter. However, for these calculations to be performed, the data must be transferred to the graphics card chip, and there can be several bottlenecks in this process.

the bandwidth of video memory, the bandwidth of the P, the speed of the processor. If this happens, then the learning speed or the speed of inference will be limited by this parameter, the number that characterizes the performance of the video chip. It is also worth mentioning that this is tied to the bit depth of numbers, such as float 32, float 16, and others. For large language models, you usually use float 16 or quantization. Therefore, this parameter should be divided by bit depth and the performance should be evaluated for a specific bit depth. 

What graphics cards did you use for your training? Well, I have my own server with three 3090 graphics cards. I’m planning to buy more soon, and I’ll be working on training them. Before building my server, I used cloud servers because it’s convenient to rent graphics cards by the hour. I have a large video on my channel comparing the main graphics cards available in the cloud, and I conducted these tests for...

To select the right graphics cards for your server, you can see their costs. There are both gaming graphics cards like the 3090 and 490, as well as server-level options like the H1 and A1. After conducting tests in the cloud using these graphics cards, I concluded that gaming-level graphics cards offer a better price-to-performance ratio for my tasks. If your goal is to train or run neural networks, I also recommend trying different graphics card options to compare training speed or inference speed on the required neural network with specific graphics cards and their quantities. After that, you can draw conclusions and choose a specific setup to either purchase or rent for an extended period. This will directly affect both training speed and inference speed. Additionally, if you want to estimate approximately how many GPU hours you need to train your model, knowing the size of your dataset and the number of parameters in the model you plan to use will be helpful.

Where in the model card do I indicate the training graph on a specific graphics processor? In my videos, I show which graphics processor I used on the platform, and in the model card, you can see the training graph, the number of training epochs, and the time spent. If you rent a server for a long time, like a month or two, there are significant discounts, which can help save on computing costs. Finally, the amount of memory is probably the most obvious parameter of any graphics card. Why is it important, especially for training? As I mentioned, training requires fitting the model weights in Half Precision (FP16) into video memory. It is crucial for the entire model to fit in video memory. There are some frameworks, such as Lama CPPM Studio, that allow keeping part of the weights in RAM, but this greatly affects the speed of the neural network's performance. Therefore, video memory must be sufficient for the entire neural network and all weights to fit in memory, leaving additional space for...

Контекст размера или значения K в кешировании зависит от того, сколько запросов может обрабатываться параллельно. Давайте я приведу некоторые цифры. Лама 70B, как уже упомянул Руслан, имеет 70 миллиардов весов. Если запускать в полной точности, то есть FP16, требуется примерно 140 ГБ видеопамяти. Но если сравнивать с большинством видеокарт, таких как H100 или A1, у них по 80 ГБ, или же с игровыми решениями, у которых максимум 24 ГБ, как у 3090 и 4090, то становится понятно, что на одной карте в такой точности запустить нейросеть не получится. Поэтому нейросети запускают либо на нескольких картах, либо используют квантизацию. При квантизации Q8, то есть 8 бит на один вес, нейросеть будет занимать 70 ГБ. Я у себя запускаю нейросеть с AWQ квантизацией, это квантизация 4 бита, и с AWQ квантизацией нейросеть занимает примерно 38 ГБ видеопамяти. Таким образом, Лама 70B с квантизацией в 4 бита может быть запущена уже на двух видеокартах 3090.

or 4090, and there will be 10 GB of context size available from this free memory, which also depends on the speed of operation. This is because, as a rule, neural networks are used by a large number of users, and the chip usually does not develop further. This is a continuation of the idea of a regular batch, only applied to the information of large language models for predicting the next token. For example, we have 10 requests, each with its own length; they are combined, and in one batch, the next token is predicted for several requests at once. In order to implement such a scheme, it is necessary for all these requests and contexts to be located within the video memory. According to the benchmarks I conducted, for example, graphics cards 3090 and 4090 are already hitting the memory limit. If there are two graphics cards, then 10 GB is already insufficient to hold all the requests that one wants to process. As a result, it turns out that the chip is underutilized. Another important parameter for training speed is to ensure that these batches are constantly transferred from...

Видеопамять и процессор работают максимально быстро, но это зависит от пропускной способности и производительности памяти видеокарты. Этот параметр становится критически важным при небольшом количестве запросов, что часто актуально для одиночных запросов. Например, когда пользователь запускает локально небольшую модель, и ему генерируется ответ. В подобном сценарии использования акцент делается на скорость видеопамяти, потому что все веса передаются, а запросов у нас всего один. Данных для вычислений относительно мало, и передача из процессора с диска практически отсутствует. Поэтому чип быстрый, но для загрузки весов требуется время. А чтобы предсказать следующий токен в единичном запросе, нужно передать все веса на видеокарту. Именно в этом месте возникает ограничение скорости, и квантизация сильно помогает в подобном сценарии. Если, например, у вас была квантизация 8 бит, а стала 4 бита, то весов становится в два раза меньше.

It is necessary to transfer data to the graphics card from the video memory, which can lead to approximately double the acceleration. The next limiting parameter is the PCI Express bus. Currently, five generations of PCI Express have been released. For example, you might see PCI Express 4 or 5, which refers to the generation of PCI Express. This standard has been developed in such a way that each subsequent generation doubles the bandwidth of the PCI Express bus. There are also different numbers of lanes that can be connected to the graphics card: x16, x8, x4, x2, and x1. The more lanes that are connected to the graphics card, the faster the bandwidth will be, which is directly proportional. For instance, if we take a PCI Express 3.0 connection with x16, the bandwidth will approximately correspond to PCI Express 4.0. In the context of tasks, the bandwidth has secondary importance. According to my tests, this is crucial during the initial query when all input tokens need to be processed. However, during token generation, the bandwidth can have a minor significance, especially when using multiple graphics cards.

Во многих случаях вся экспрессия является ограничивающим фактором, потому что все градиенты с одной видеокарты нужно передать на другую. В частности, стоит упомянуть технологии NVIDIA, такие как NV Switch и NVLink, которые по-разному называются. Данная технология поддерживается, как правило, серверами видеокартами, в то время как в игровых видеокартах поддержка NVLink закончилась на 3000-м поколении. То есть 3090 имеет NVLink, а 4090 уже не имеет этого подключения. Поэтому, если нужно тренировать большие модели, это как правило делают на серверных видеокартах H100 и A100, которые подключены к высокоскоростной шине NV. Потому что на данный момент Pess медленнее, чем проприетарное решение от NVIDIA. 

Ещё нужно рассказать о процессоре. Именно центральный процессор в ряде случаев может являться ограничивающим фактором обучения. На процессоре, как правило, готовятся данные, и после этого они передаются на видеокарту. В задачах инса процессоры имеют второстепенную роль. Туда подаются текстовые данные.

Language models transmit text to the graphics card, and there, the situation becomes more complex with images because images are usually processed by the CPU. If we need to perform operations on images, the image is loaded either from disk or over the network. Then, the central processor decompresses it, usually in formats like GP, PNG, or other formats. This compressed image format is transformed into a matrix of the required dimensions, and this data is then sent to the graphics card. At this stage, the processor can be a limiting factor. When it comes to training with images, augmentation is typically used. Augmentation is a process of making small changes to the data so that the model does not memorize the original image. For example, if we have a dataset of cat images that is relatively small, we need to ensure that the model doesn't just memorize these images. Therefore, we apply augmentation techniques, such as slight changes in brightness or small shifts of the image to the left or right.

scaling up and down, cropping, changing gamma, and a number of other parameters. These are small changes that the human eye can perceive; for example, the same cat might appear slightly darker, or its position might have shifted a bit. However, for a neural network, this represents a completely different image because the matrix has fundamentally changed, and it learns to draw conclusions based on the augmented image. 

At one of my workplaces, we had such a scenario. There was a server with two graphics cards and an eight-core processor. We were running two training processes, specifically focused on image augmentation, and the emphasis was on the processor. When we launched training on one graphics card, the processor was loaded to about six cores. When we simultaneously launched a different variant on the second graphics card, the processor was fully utilized with all eight cores, and the training of both networks was happening more slowly. Therefore, it is essential to consider parameters like these.

Performance across all cores. Next, I want to talk about the support for new instructions such as FP8 and FP4 in NVIDIA graphics cards, specifically the H100 and newer H200 models. Support for new instructions like FP8 has been introduced primarily for neural networks. As I mentioned, neural networks are usually run externally in a lower dimensionality. For example, the H100 supports operations with the FP8 format. If you load a network in the FP8 format, which is 8 bits, the H100 graphics card can natively support these instructions. However, if we consider the previous generation of graphics cards, such as the A100, 3090, or earlier versions, there is no support for this instruction, and it is not natively supported. For instance, when quantizing models like AWQ or GG in Q8, we first take the weights, convert them to the FP16 format, and then all operations occur in the FP16 format on the graphics chip. The result is then returned, meaning that the previous generation operates in this manner.

Сценарии будут медленнее, чем новейшие поколения, такие как H1 и H200 видеокарт, и на них есть нативная поддержка этих инструкций. Это стоит учитывать для тренировки нейросетей. Задача тренировки нейросетей в FP8 редко применяется, чаще всего используется FP16 для тренировки модели. Но я думаю, что с ростом моделей и увеличением количества видеокарт, которые поддерживают нативно FP8 и более низкие размерности, такой формат будет более популярен. Университеты тоже будут тренироваться в FP8 и FP4 форматах, и видеокарты нового поколения будут выгодно отличаться от предыдущих поколений. Я думаю, помимо поддержки форматов на уровне железа, еще значение имеет наличие библиотек, которые поддерживают способ обучения. Наверное, чтобы облегчить вход для инженеров, NVIDIA выпустила свой фреймворк Transform Engine, с помощью которого можно обучать в FP8 на H100. Но, конечно, большинство инженеров сейчас сосредоточено на стеке Henf Transformers, где люди привыкли обучать.

The last question is about the disk and how it affects training. Neural networks typically store model checkpoints and data on the disk, and in rare cases, the disk can become a bottleneck. Let's consider a scenario where this might be important. For instance, if we are training a small neural network, perhaps for quick inference on a mobile phone or embedded devices like Jetson Nano or Jetson Orion, the size of the network is small, but we want to train it on a large amount of data. In this case, we have a large dataset of images that need to be loaded from the disk, augmented, and then sent to the graphics card. If we have a sufficiently small neural network and a fast graphics card, we may encounter limitations due to the speed of the disk. This is particularly relevant for older disks with SATA interfaces.

500 MB per second is the reading speed, and just in this interface, there will be more modern drives with PCI Express connectivity. These drives are either inserted directly into the PCI Express slot or connected via interface 2. In that case, this issue becomes less relevant. If we consider, for example, the 70B, the speed of the drive will be determined by how quickly the model starts and loads into the video memory. For this process, it is necessary to read all the weights from the drive. I would like to remind you that even when considering quantization at 140 GB, these 140 GB need to be read and transferred to the graphics card's memory, and it is here that the drive will be a limiting factor. As for ISA, it is usually not that important to wait for the start to finish; what matters more is how it operates. Therefore, we can even use older drives. Today's release was in a Colab format. I recommend Ruslan's channel, which will be linked in the description of this video. Go check it out; he talks about training and also offers a different perspective on it.

I would also like to remind you that the sponsor of this video is Cloud, a company that provides servers with all these graphics cards. You can take advantage of their services; the link will also be in the description of this video. This was Vitaly Kuliev. Goodbye, everyone!

Топ советов по выбору видеокарты для машинного обучения и инференса нейросетей(LLM)?

Scripts