The performance of the Diablo GPU, as well as the hardware, has improved significantly, and the software has also advanced a lot. There was a time when Dr. Lisa Su announced whether AMD could truly penetrate NVIDIA's market, which coincided with her 10th anniversary as CEO of AMD. The stock price has increased by over 100 times. 

Hello everyone, this is Andar Engineering. On October 10th in San Francisco, Dr. Lisa Su presented various products. AMD introduced Advancing AI 2024, a new server CPU, and shared its roadmap. During that week, a new AI GPU, the H200, emerged, claiming to outperform NVIDIA's H200, attracting a lot of attention. This product is the MI 325X, and even though NVIDIA still dominates the AI training and inference market, we took some time to examine the significance of AMD's recent release and review various specifications.

I will take a look at this. In fact, from the perspective of AI GPUs, there have been evaluations stating that, when viewed from the standpoint of a single GPU, it seems to outperform NVIDIA GPUs. In this presentation, Satya Nadella from Microsoft appeared, and there was a scene showing them engaging in a discussion. However, as I always mention in my videos, it's not just one thing that matters. AI GPUs operate on such an enormous scale that we refer to them as hyperscalers. To conduct L&M learning at this scale, simply having one GPU is not enough. It requires connecting dozens or even hundreds of GPUs together, enabling them to process data collectively through networking. It would be beneficial to examine the content presented today from this crucial perspective. No matter how much NVIDIA holds a nearly monopolistic position, AMD...

Is it enough to just stay still? So, AMD released the MI 300X last year, and this time they have introduced the MI 300X, which can be seen as an expansion of the previous MI 300X memory. It has the same CDNA 3 architecture. Next year, in 2025, NVIDIA's Black will be released, and in response to that, the MI 350 will be based on the new CDNA 4 architecture. They have presented a roadmap for launching products based on this fourth architecture. In fact, AMD has been establishing a roadmap for annual releases like this, which has led to a change in the development cycle for NVIDIA's H00 and H200 from the original two years to one year. However, AMD has consistently promised to release GPUs that will respond accordingly. What you are seeing now is the AMD Instinct MI 325X chip. Looking at the major server companies, we have Dell and...

HPN, Lenovo, and Supermicro are collaborating with large companies like NVIDIA. In fact, hyperscalers such as Apple, Meta, and Microsoft are inevitably purchasing and using MBA products, but they need to consider a dual vendor strategy with AMD. It seems that they are consistently collaborating in this manner. Production is expected to start in the fourth quarter, so from the first quarter of next year, it is predicted that various solutions can be supported based on the AMD Instinct MI 325X infrastructure. 

Looking at the specifications, the MI 300X utilizes the CDNA 3 architecture, and as emphasized, it uses HBM 3E memory. The maximum capacity has now increased to 256GB, allowing for a substantial amount of memory to be integrated into a single GPU, accommodating parameters and other requirements.

It seems that they are focusing on the fact that the space has significantly increased. And as you can see here, the bandwidth has also increased to 6 terabits per second, which is 1.3 times higher. In terms of floating point, it is 1.3 for FP16, and for FP8, it is double, reaching 2.62. So, it looks impressive as they present this information. Dr. Lisa Su is now showcasing the signature. Compared to previous versions, it is certainly improved, targeting a level of performance for individual GPUs that is quite comfortably higher. Of course, the H200 product is already available, so this isn't something that can be compared one-to-one, and we need to consider these factors together, which I will explain later in the video. However, this also fundamentally supports floating point 16 and floating point 8, while NVIDIA also offers support for FP6 and FP4, which means 6-bit and 4-bit.

As Bit also supports this, it is preparing to handle up to 9.2 petaflops. Right now, there are many attempts to significantly reduce the pricing in the AI inference market, making it much cheaper. This is why they are trying to support much smaller models and more affordable options. AMD is also aiming to lower the parameters, using four or six bits per parameter, which will reduce the amount of computation needed. This is similar to what was discussed regarding using four bits. It seems they are preparing accordingly. The platform based on the MI 3X GPU will be set up with eight units, similar to how NVIDIA also combines eight units in their H200 HGX platform. Comparing these two platforms will show differences in memory.

The capacity is 2TB, which is about 1.8 times higher. As you can see, computing capabilities such as floating point 16 and floating point 8 will also increase linearly. So, it can be said that it increases by eight times and 1.3 times compared to NVIDIA. Additionally, the bandwidth is at 48TB per second, which is 1.3 times higher than the previous level. When comparing it like this, it shows that the performance is higher than the NVIDIA H200 HGX, which refers to eight combined GPUs. This comparison is made between one GPU and eight GPUs, claiming that there is a maximum of 1.4 times improvement in inference performance. The reason for discussing this in various ways is that in the AI market, people who buy AI GPUs, of course, are purchasing a lot of hyperscale systems. However, there are also companies that cannot buy many.

Especially in our country, we can’t buy many GPUs. In fact, hyper scalers entering Asia purchase tens of thousands of units, but in our country, when they come in, it’s at a level of about 1,000 units at a time. Looking at the situation in our country, we receive much smaller quantities than that. Therefore, the utilization will vary depending on the amount we can receive. Unlike tech companies, startups will take smaller quantities, so the performance of these eight units can be an important indicator to some extent. So, rather than just saying that the DIA has done well, we also need to consider the performance at the platform level. What was impressive was that AMD presented a platform related to networking, whereas NVIDIA offers its own proprietary InfiniBand.

To use NVIDIA, you need to use this, and you are building the networking part. We are using our specifications with MV links and MV switches to bring everything together quickly, and the performance is incredibly good. It's overwhelming. However, for the next generation of AI networking, AMD has been contemplating these aspects, particularly in the networking area. There is the DPU, commonly referred to as BlueField by NVIDIA, which can be viewed as a Data Processing Unit. They are saying they will respond by utilizing this programmable DPU. If we broadly categorize AI networking, we can divide it into two parts: the front end and the back end. The front end is responsible for delivering data and information to the AI cluster, while the back end manages data transmission between accelerator clusters. So, what happens in the background...

The tasks involve managing data transfers within the accelerator while moving back and forth between the server. This is referred to as the backend. Delivering data and information to the AI cluster is considered the frontend. AMD has divided the frontend and backend, creating a corresponding table for this. In the frontend network, data is quickly collected, allowing for the acceleration of storage. This role is preemptively fulfilled, facilitating rapid networking. What does networking entail? Essentially, when data is collected, it is processed in between and disseminated quickly. This is how the frontend and backend are divided, and you can understand it as creating a table for correspondence. So, with the processing done by the DPU, it is expanded to allow for Ethernet scalability, and NVIDIA...

The InfiniBand has a separate specification called ROC version 2, or ROC V2. I previously summarized this in a video. When using Ethernet, it is claimed that the overall cost is 50% lower compared to InfiniBand. From a scalability perspective, InfiniBand can expand up to 48,000 GPUs, while Ethernet can currently scale up to one million, indicating much greater potential. Along with this, they announced the P4 Engine, which can transmit 400 GB per second and can connect up to five million connections per second. This allows for programming capabilities while simultaneously providing services. It seems that they are preparing something to compete with AMD and NVIDIA in the networking area. However, using ROC V2 seems to promote a more open ecosystem.

Using Ethernet, we are accessing the designated point. NVIDIA operates in a closed manner, insisting that we use only their technology, while AMD offers greater scalability. As I mentioned earlier regarding front-end and back-end, we use Solana or similar technologies for the front end and Polara for the back end. The role of the DPU, such as Salina, can be thought of as a receptionist at a delivery hub center. Packages arrive at the hub center, and the receptionist manages the incoming parcels. NVIDIA's BlueField performs that role in the network; as data flows in, it manages security while checking and appropriately distributing various data to send out to the AI cluster. This role is crucial as parcels must pass through this employee to be processed.

It would be good to understand that this relates to the architecture of frontend systems or DPU. In the case of Collara, it has been taken over, but you can think of it as an automatic branching system at a tech-back center. Once the data is received, it will automatically distribute it along the designated routes. So, within the GPU, AI accelerators need to transfer data among themselves. This means that the backend servers can communicate back and forth automatically. You can think of it as a technology that allows for seamless connections, similar to NVIDIA's MVLink or MVSwitch, enabling data transfer. This involves having separate processors to ensure that they can efficiently distribute data. Therefore, it reflects not only the performance of the GPUs themselves but also NVIDIA's ongoing efforts to keep up with and respond to technological advancements.

The platform is expected to be commercialized in the first half of next year. Additionally, NVIDIA introduced its software platform, ROCm, which competes with CUDA. This solid software platform, built by NVIDIA over the past 18 years, was discussed in an interview I had with Lee Kyung-il, the CEO of Saltlux. He mentioned that NVIDIA is a hardware company, but more accurately, it is a software company. To put it precisely, it is a company that designs hardware chips and has a strong presence in software. He pointed out that no matter how good AMD's hardware is, if the software does not support it, people will not use it. Many who have learned AI have started with CUDA, and although ROCm is still lacking, it continues to evolve as a powerful new open software stack.

We are continuously expanding our technology to provide functionality. This is in response to NVIDIA, as we aim to remain an open community. We are supporting various models to be readily usable within frameworks like Stable Diffusion 3, Meta's Llama, and Hugging Face through software development. We have also updated ROCm 6.2, which is well optimized for AMD hardware, and we are committed to advancing open software as an open community. When comparing the actual performance of 6.0 and 6.1, there has been significant improvement. For instance, in the case of mixed precision, performance has improved by more than double. For Llama 3.1, with 70 billion and 80 billion parameters, the performance has increased from 2600 to 28 times better.

On average, performance has increased by about 2.4 times. This means that the software has developed significantly and has been well-produced. The performance of the Diablo GPU is also good compared to the H200; both the hardware and software have advanced greatly. So, can AMD really encroach on NVIDIA's market? I still think it's quite tough, especially regarding learning, as NVIDIA's platform, which involves various GPUs working together, is extremely powerful. AMD's Infinity Fabric supports up to 896 GB per second and is improving rapidly, but NVIDIA is currently providing support at twice that speed. This is true for MVLink as well, and when it comes to linking various platforms, even if individual GPU performance is slightly better, there are still many challenges.

Companies prefer NVIDIA over AMD, and they have invested heavily in it. The CUDA ecosystem is quite strong. If they want to develop programs based on CUDA and expand from there, they would want to continue extending the infrastructure that is based on CUDA. If they were to switch to AMD, they would need to hire additional personnel, which is a significant consideration. Although computing technologies have improved significantly lately, making adjustments to existing systems still requires time and effort. The AR market is moving very quickly, and I think there is a burden associated with that. Nevertheless, the reason why biotech companies and server firms are collaborating with AMD is that they need to continue developing these capabilities. This way, the prices can remain competitive.

AMD might still face challenges, but it is catching up quite well. There are still many areas where it hasn't fully developed. As for next year, it seems like all the products have been sold out, and the talk of being overwhelmed is still present. However, it appears that we need to keep pushing forward. 

There was a time when AMD's CPUs were struggling due to the Bulldozer architecture, which led to some tough years. The announcement was made around the time when Dr. Lisa Su had just completed ten years at AMD, which is quite significant. The company's stock price has increased over 100 times, marking a truly dramatic development. 

In the server space, AMD had hardly made any inroads against Intel, but now it has captured over 30% of the market share. I believe the new Intel CPUs will not outperform AMD. Over the past ten years, so much has changed. Currently, we are in the early stages of the AI market, where the infrastructure is being laid out.

I think it is very meaningful that AMD is keeping up with the times. Ultimately, in the tech industry, competition is essential for further development, right? Given that CPUs have an international culture, it would be good to keep an eye on AMD's future investments related to AI as well. I would like to introduce you to the Andal Engineering Instagram. It shares brief content that is not covered on YouTube, providing updates and fun information. Please follow us, and I would appreciate it. This has been Anbal Engineering.

AMD 리사수, NVIDIA GPU 대응 HBM3E 기반 Instinct MI325X 발표 | CUDA, infiniband, NVLink 대응 SW 및 네트워킹 플랫폼 업그레이드

공학박사 '에러'가 전하는
최신 공학/테크/IT/신기술

문의 : Unrealtech2021@gmail.com

AMD 리사수, NVIDIA GPU 대응 HBM3E 기반 Instinct MI325X 발표 | CUDA, infiniband, NVLink 대응 SW 및 네트워킹 플랫폼 업그레이드

AMD 리사수, NVIDIA GPU 대응 HBM3E 기반 Instinct MI325X 발표 | CUDA, infiniband, NVLink 대응 SW 및 네트워킹 플랫폼 업그레이드

Scripts