How many calculations do you think your graphics card performs every second while running video games with incredibly realistic graphics? Maybe 100 million? Well, 100 million calculations a second is what’s required to run Mario 64 from 1996.

We need more power. Maybe 100 billion calculations a second? Well, then you would have a computer that could run Minecraft back in 2011.

In order to run the most realistic video games such as Cyberpunk 2077, you need a graphics card that can perform around 36 trillion calculations a second.

This is an unimaginably large number, so let’s take a second to try to conceptualize it. Imagine doing a long multiplication problem once every second. Now let’s say everyone on the planet does a similar type of calculation but with different numbers.

To reach the equivalent computational power of this graphics card and its 36 trillion calculations a second, we would need about 4,400 Earths filled with people, all working together and completing one calculation each every second.

It’s rather mind boggling to think that a device can manage all these calculations, so in this video, we’ll see how graphics cards work in two parts.

First, we’ll open up this graphics card and explore the different components inside, as well as the physical design and architecture of the GPU, or graphics processing unit.

Second, we’ll explore the computational architecture and see how GPUs process mountains of data, and why they’re ideal for running video game graphics, Bitcoin mining, neural networks, and AI.

So, stick around and let’s jump right in.

This video is sponsored by Micron, which manufactures the graphics memory inside this graphics card. Before we dive into all the parts of the GPU, let’s first understand the differences between GPUs and CPUs.

Inside this graphics card, the Graphics Processing Unit, or GPU, has over 10,000 cores. However, when we look at the CPU, or Central Processing Unit, that’s mounted to the motherboard, we find an integrated circuit or chip with only 24 cores.

So, which one is more powerful? 10 thousand is a lot more than 24, So you would think the GPU is more powerful, however, it’s more complicated than that.

A useful analogy is to think of a GPU as a massive cargo ship and a CPU as a jumbo jet airplane.

The amount of cargo capacity is the amount of calculations and data that can be processed, and the speed of the ship or airplane is the rate at which how quickly those calculations and data are being processed.

Essentially, it’s a trade-off between a massive number of calculations that are executed at a slower rate versus a few calculations that can be performed at a much faster rate.

Another key difference is that airplanes are a lot more flexible since they can carry passengers, packages, or containers and can take off and land at any one of tens of thousands of airports.

Likewise CPUs are flexible in that they can run a variety of programs and instructions. However, giant cargo ships carry only containers with bulk contents inside and are limited to traveling between ports.

Similarly, GPUs are a lot less flexible than CPUs and can only run simple instructions like basic arithmetic. Additionally GPUs can’t run operating systems or interface with input devices or networks.

This analogy isn’t perfect, but it helps to answer the question of “which is faster, a CPU or a GPU?”. Essentially if you want to perform a set of calculations across mountains of data, then a GPU will be faster at completing the task.

However, if you have a lot less data that needs to be evaluated quickly than a CPU will be faster.

Furthermore, if you need to run an operating system or support network connections and a wide range of different applications and hardware, then you’ll want a CPU.

We’re planning a separate video on CPU architecture, so make sure to subscribe so you don’t miss it, but let’s now dive into this graphics card and see how it works. In the center of this graphics card is the printed circuit board or PCB, with all the various components mounted on it, and we’ll start by exploring the brains which is the graphics processing unit or GPU.

When we open it up, we find a large chip or die named GA102 built from 28.3 billion transistors. The majority of the area of the chip is taken up by the 
processing cores, which have a hierarchical organization.

Specifically, the chip is divided 
into 7 Graphics Processing Clusters, or GPCs, and within each processing cluster are 12 
streaming multiprocessors, or SMs.

Next, inside each of these streaming multiprocessors 
are 4 warps and 1 ray tracing core, and then, inside each warp are 32 CUDA or shading cores, and 
1 tensor core.

Across the entire GPU are 10,752 CUDA cores, 336 Tensor Cores, and 84 Ray Tracing 
Cores.

These three types of cores execute all the calculations of the GPU, and each has a different 
function.

CUDA cores can be thought of as simple 
binary calculators with an addition button, a 
multiply button, and a few others, and are used the most when running video games.

Tensor cores 
are matrix multiplication and addition calculators 
and are used for geometric transformations 
and working with neural networks and AI. And ray tracing cores are the largest but the fewest 
and are used to execute ray tracing algorithms.

Now that we understand the computational 
resources inside this chip, one rather interesting fact is that the 3080, 3090, 3080 Ti, and 3090 Ti 
graphics cards all use the same GA102 chip design for their GPU.

This might be counterintuitive 
because they have different prices and were 
released in different years, but it’s true. 
So, why is this? Well, during the manufacturing process, sometimes patterning errors, dust 
particles, or other manufacturing issues 
cause damage and create defective areas of the 
circuit.

Instead of throwing out the entire chip because of a small defect, engineers find the 
defective region and permanently isolate and 
deactivate the nearby circuitry.

By having a GPU 
with a highly repetitive design, a small defect in one core only damages that particular streaming 
multiprocessor circuit and doesn’t affect the 
other areas of the chip.

As a result, these chips 
are tested and categorized or binned according to the number of defects.

The 3090 Ti graphics cards 
have flawless GA102 chips with all 10,752 CUDA cores working properly.

The 3090 has 10,496 
cores working, the 3080 Ti has 10,240, and the 3080 has 8,704 CUDA cores working, which is equivalent 
to having 16 damaged and deactivated streaming   multiprocessors.

Additionally, different graphics 
cards differ by their maximum clock speed and the quantity and generation of graphics memory that 
supports the GPU, which we’ll explore in a little   bit.

Because we’ve been focusing on the physical 
architecture of this GA102 GPU chip, let’s zoom into one of these CUDA cores and see what it looks 
like.

Inside this simple calculator is a layout   of approximately 410,000 transistors. This 
section of 50,000 transistors performs the operation of A times B plus C, which is called 
fused multiply and add, or FMA, and is the most   common operation performed by graphics cards.

Half of the CUDA cores execute FMA using 32-bit floating-point numbers, which is essentially 
scientific notation, and the other half   of the cores use either 32-bit integers or 32-bit 
floating point numbers.

Other sections of this core accommodate negative numbers and perform 
other simple functions like bit-shifting and bit   masking, as well as collecting and queueing 
the incoming instructions and operands, and then accumulating and outputting the results.

As a result, this single core is just a simple   calculator with a limited number of functions. 
This calculator completes one multiply and one add operation each clock cycle, and therefore with this 
3090 graphics cards and its 10,496 cores and 1.7 gigahertz clock, we get 35.6 trillion calculations 
a second.

However, if you’re wondering how the GPU handles more complicated operations like division, 
square root, and trigonometric functions, well,   these calculator operations are performed by 
the special function units, which are far fewer as only four of them can be found in each streaming multiprocessor.

Now that we have an understanding of what's inside a single core, let's zoom out and take a look at the other sections of the GA102 chip.

Around the edge, we find 12 graphics memory controllers, the NVLink Controllers, and the PCIe interface. On the bottom is a 6-megabyte Level 2 SRAM Memory Cache, and here's the Gigathread Engine, which manages all the graphics processing clusters and streaming multiprocessors inside.

Now that we've explored this GA102 GPU's physical architecture, let's zoom out and take a look at the other parts inside the graphics card.

On this side are the various ports for the displays to be plugged into, on the other side is the incoming 12-volt power connector, and then here are the PCIe pins that plug into the motherboard.

On the PCB, the majority of the smaller components constitute the voltage regulator module, which takes the incoming 12 volts and converts it to 1.1 volts and supplies hundreds of watts of power to the GPU.

Because all this power heats up the GPU, most of the weight of the graphics card is in the form of a heat sink with four heat pipes that carry heat from the GPU and memory chips to the radiator fins, where fans then help to remove the heat.

Perhaps some of the most important components, aside from the GPU, are the 24 gigabytes of graphics memory chips, which are technically called GDDR6X SDRAM and were manufactured by Micron, which is the sponsor of this video.

Whenever you start up a video game or wait for a loading screen, the time it takes to load is mostly spent moving all the 3D models of a particular scene or environment from the solid-state drive into these graphics memory chips.

As mentioned earlier, the GPU has a small amount of data storage in its 6-megabyte shared Level 2 cache, which can hold the equivalent of about this much of the video game's environment.

Therefore, in order to render a video game, different chunks of scene are continuously being transferred between the graphics memory and the GPU.

Because the cores are constantly performing tens of trillions of calculations a second, GPUs are data-hungry machines and need to be continuously fed terabytes upon terabytes of data.

Thus, these graphics memory chips are designed kind of like multiple cranes loading a cargo ship at the same time. Specifically, these 24 chips transfer a combined 384 bits at a time, which is called the bus width, and the total data that can be transferred, or the bandwidth, is about 1.15 terabytes a second.

In contrast, the sticks of DRAM that support the CPU only have a 64-bit bus width and a maximum bandwidth closer to 64 gigabytes a second.

One rather interesting thing is that you may think that computers only work using binary, ones and zeros.

However, in order to increase data transfer rates, GDDR6X and the latest graphics memory, GDDR7, send and receive data across the bus wires using multiple voltage levels beyond just 0 and 1.

For example, GDDR7 uses three different encoding schemes to combine binary bits into ternary digits or PAM-3 symbols with voltages of 0, 1, and negative 1.

Here's the encoding scheme on how three binary bits are encoded into two ternary digits, and this scheme is combined with an 11 bit to seven ternary digit encoding scheme resulting in sending 276 binary bits using only 176 ternary digits.

The previous generation, GDDR6X, which is the memory in this 3090 graphics card, used a different encoding scheme, called PAM-4, to send two bits of data using four different voltage levels.

However, engineers and the graphics memory industry agreed to switch to PAM-3 for future generations of graphics chips in order to reduce encoder complexity, improve the signal-to-noise ratio, and improve power efficiency.

Micron delivers consistent innovation to push the boundaries on how much data can be transferred every second and to design cutting-edge memory chips.

Another advancement by Micron is the development of HBM, or high-bandwidth memory, that surrounds AI chips. HBM is built from stacks of DRAM memory chips and uses TSVs or through silicon vias, to connect this stack 
into a single chip, essentially forming a cube 
of AI memory.

For the latest generation of high 
bandwidth memory, which is HBM3E, a single cube can have up to 24 to 36 gigabytes of memory, thus 
yielding 192 gigabytes of high-speed memory around  the AI chip.

Next time you buy an AI accelerator 
system, make sure it uses Micron’s HBM3E, which uses 30% less power than the competitive products.

However, unless you’re building an AI data center, 
you’re likely not in the market to buy one 
of these systems which cost between 25 to 40 thousand dollars and are on backorder for a 
few years.

If you’re curious about high bandwidth 
memory, or Micron’s next generation of graphics 
memory, take a look at one of these links in the description.

Alternatively, if designing the next 
generation of memory chips interests you, Micron 
is always looking for talented scientists and 
engineers to help innovate on cutting edge chips, and you can find out more about working for Micron 
using this link.

Now that we’ve explored many of 
the physical components inside this graphics card 
and GPU, let’s next explore the computational architecture and see how applications like video 
game graphics and bitcoin mining run what’s called 
“embarrassingly” parallel operations.

Although 
it may sound like a silly name, embarrassingly parallel is actually a technical classification 
of computer problems where little or no effort is 
needed to divide the problem into parallel tasks, 
and video game rendering and bitcoin mining easily fall into this category.

Essentially, GPUs solve 
embarrassingly parallel problems using a principle 
called SIMD, which stands for single instruction 
multiple data where the same instructions or steps are repeated across thousands to millions of 
different numbers.

Let’s see an example of how 
SIMD or single instruction multiple data is used 
to create this 3D video game environment. As you may know already, this cowboy hat on the table is 
composed of approximately 28 thousand triangles 
built by connecting together around 14,000 
vertices, each with X, Y, and Z coordinates.

These vertex coordinates are built using a coordinate system called model space with the origin of 0, 0, 0 being at the center of the hat. To build a 3D world, we place hundreds of objects, each with their own model space into the world environment, and in order for the camera to be able to tell where each object is relative to other objects, we have to convert or transform all the vertices from each separate model space into the shared world coordinate system, or world space.

So, as an example, how do we convert the 14,000 vertices of the cowboy hat from model space into world space?

Well, we use a single instruction which adds the position of the origin of the hat in world space to the corresponding X, Y, and Z coordinate of a single vertex in model space.

Next, we copy this instruction to multiple data, which is all the remaining X, Y, and Z coordinates of the other thousands of vertices that are used to build the hat. Next, we do the same for the table and the rest of the hundreds of other objects in the scene, each time using the same instructions but with the different objects’ coordinates in world space, and each objects’ thousands of vertices in model space.

As a result, all the vertices and triangles of all the objects are converted to a common world space coordinate system, and the camera can now determine which objects are in front and which are behind.

This example illustrates the power of SIMD, or single instruction multiple data, and how a single instruction is applied to 5,629 different objects with a total of 8.3 million vertices within the scene, resulting in 25 million addition calculations.

The key to SIMD and embarrassingly parallel programs is that every one of these millions of calculations has no dependency on any other calculation, and thus all these calculations can be distributed to the thousands of cores of the GPU and completed in parallel with one another.

It's important to note that vertex transformation from model space to world space is just one of the first steps of a rather complicated video game graphics rendering pipeline, and we have a separate video that delves deeper into each of these other steps.

Also, we skipped over the transformations for the rotation and scale of each object, but factoring these values is a similar process that requires
additional SIMD calculations.

Now that we have a simple understanding of SIMD, let's discuss how this computational architecture matches up with the physical architecture.

Essentially, each
instruction is completed by a thread, and this thread is matched to a single CUDA core. Threads are bundled into groups of 32 called warps, and the same sequence of instructions is issued to
all the threads in a warp.

Next, warps are grouped into thread blocks which are handled by the streaming multiprocessor. And then finally thread blocks are grouped into grids, which are computed
across the overall GPU.

All these computations are managed or scheduled by the Gigathread Engine, which efficiently maps thread blocks to the available streaming multiprocessors.

One important
distinction is that within SIMD architecture, all 32 threads in a warp follow the same instructions and are in lockstep with each other, kind of like a phalanx of soldiers moving
together.

This lockstep execution applied to GPUs up until around 2016. However, newer GPUs follow a SIMT architecture, or single instruction multiple threads.

The difference between SIMD and SIMT is
that while both send the same set of instructions to each thread, with SIMT, the individual threads don't need to be in lockstep with each other and can progress at different rates.

In technical jargon, each thread is given its own program counter. Additionally, with SIMT, all the threads within a streaming multiprocessor use a shared 128 kilobyte L1 cache, and thus data that's
output by one thread can be subsequently used by a separate thread.

This improvement from SIMD to
SIMT allows for more flexibility when encountering warp divergence via data-dependent conditional branching and easier reconvergence for the threads to reach the barrier synchronization.

Essentially,
newer architectures of GPUs are more flexible and efficient, especially when encountering branches in code. One additional note is that although you may think that the term warp is derived from
warp drives, it actually comes from weaving and specifically the Jacquard Loom.

This loom from 1804 used programmable punch cards to select specific threads out of a set to weave together intricate patterns.

As fascinating as looms are, let’s move on. The final topics we’ll explore are Bitcoin mining, tensor cores, and neural networks.

But first, we’d like to ask you to like this video, write a quick comment below, share it with a colleague, friend, or on social media, and subscribe if you haven’t already.

The dream of Branch Education is to make free and accessible, visually engaging educational videos that dive deeply into a variety of topics on science, engineering, and how technology works, and then to combine multiple videos into an entirely free engineering curriculum for high school and college students.

Taking a few seconds to like, subscribe, and comment below helps us a ton!

Additionally, we have a Patreon page with AMAs and behind-the-scenes footage, and if you find what we do useful, we would appreciate any support. Thank you. So now that we’ve explored how single instruction multiple threads is used in video games, let’s briefly discuss why GPUs were initially used for mining Bitcoin.

We’re not going to get too far into the algorithm behind the blockchain and will save it for a separate episode, but essentially, to create a block on the blockchain, the SHA-256 hashing algorithm is run on a set of data that includes transactions, a timestamp, additional data, and a random number called a nonce.

After feeding these values through the SHA-256 hashing algorithm, a random 256-bit value is output. You can kind of think of this algorithm as a lottery ticket generator where you can’t pick the lottery number, but based on the input data, the SHA-256 algorithm generates a random lottery ticket number.

How do Graphics Cards Work?  Exploring GPU Architecture

How do Microchips work?  How does your smartphone camera work?  How does Bluetooth work?  At Branch Education, we dive deep into the science, engineering, and technology in our modern world using 3D animation and incredibly detailed models. 

Founder of Branch Education: Theodore J Tablante

Want to support in-depth engineering and technology education? Support us on Patreon:  http://www.patreon.com/brancheducation

Modeling is done in Blender 3.5 Cycles Render Engine, Adobe Premiere, and Reaper

How do Graphics Cards Work? Exploring GPU Architecture

Scripts