Google's TPU Development: Innovations and Industry Impact

Google's custom AI chips, especially TPUs, have revolutionized cloud computing and positioned it competitively in the AI landscape.

### Google's Custom Chips: The Evolution and Future of AI Processing
* 🖥️  At Google's Silicon Valley headquarters, a lab is dedicated to testing its latest generation of **Tensor Processing Units (TPUs)**, crucial for powering various Google services, including **search and YouTube ads**.
* 🔍  The first Trillium system, featuring **256 chips**, exemplifies Google's commitment to custom chip development, which began nearly a decade ago to **enhance voice recognition capabilities**.
* 💡  Google originally coined the term "**Tensor Processing Unit**" and has since become a leader in custom AI chips, holding a **58% market share** in this sector, while competitors like Amazon and Microsoft have only recently entered the field.
* ⚡  Despite its pioneering role in AI hardware, Google has faced criticism for **falling behind in the generative AI race**, notably with the **delayed release of its chatbot, Gemini**, which utilizes TPUs for training.
* 💧  The company has also taken significant steps to **improve energy efficiency and reduce water usage** in its data centers, implementing **direct-to-chip cooling systems** to address environmental concerns.
* 🌍  As the demand for AI capabilities grows, Google is investing heavily in chip development, including its upcoming **general-purpose CPU, Axion**, although it has been **slower to enter this market** compared to rivals.
* 🔗  Partnerships with companies like Broadcom have been essential for Google in creating **complex AI chips**, highlighting the **intricate nature of chip design and manufacturing**, particularly in light of **geopolitical risks**.
* 🚀  With the AI industry rapidly evolving, Google remains focused on **advancing its chip technology** to meet the **increasing computational demands of AI applications**, indicating a **promising future for its custom silicon initiatives**.

At this sprawling lab at Google's Silicon Valley headquarters, these racks and racks of servers aren't running workloads for Google Cloud's millions of customers. Here, for example, is the very first Trillium system that we built is a full Trillium system with 256 chips in it for racks. Or for YouTube, or the world's most dominant search engine. And what is Trillium? Trillium is our latest generation TPU. It'll be public later this year. Instead, they're running tests on its very own microchips, Tensor Processing Units, that help power it all, search. And of course, video, YouTube ads. Everything Google does has been powered in many ways by its own homegrown TPU. Now, TPUs are used to train AI models like Google's own chatbot Gemini. .. and in some big news, Apple's AI, too. Apple, actually, we found out yesterday they disclosed in a paper they're using Google-made chips.

The world sort of has this fundamental belief that all AI large language models are being trained on Nvidia. But Google took its own path here. And yet, despite being the birthplace of some foundational concepts behind generative AI, many say Google's fallen behind in the AI race. But it was the first major cloud provider to do custom AI chips. It was ten years ago, almost to the day, where we decided that to meet the needs of our users in terms of a particular application, voice recognition at the time, we needed to design custom hardware. In the years since, Amazon, Microsoft, and Meta have started making their own AI chips too. Here we're turning on the chips and the boards for the first time, making sure they're working properly to specification, debugging any issues that might come up, that sort of thing. And no media has been inside here before. First time? Yep. We went to Google headquarters for an exclusive look inside the chip lab and sat down with its top executive.

to ask why and how. Google's betting big on the expensive, complex business of custom chips. It all started in 2014 when a group at Google calculated that in order to launch upcoming voice recognition features, Google would need to double the number of computers in its data centers. Amin Vahdat, now the head of custom cloud chips, started at Google four years before that. A number of leads at the company asked the question, what would happen if Google users wanted to interact with Google via voice for just 30s a day? And how much compute power would we need to support our users? We realized that we could build custom hardware, not general-purpose hardware, but custom hardware Tensor Processing Units in this case to support that much, much more efficiently. In fact, a factor of 100 more efficiently than it would have been otherwise. What is a Tensor Processing Unit? And did you guys coin that term? We did. We coined the term Tensor Processing Unit.

We believe that it was certainly the first large-scale hardware accelerator for AI applications. There's a whole gamut of qualification and validation tests we do on, you know, power thermals functionality. You're really trying to make sure the design has enough margin so that it's going to operate well, you know, at volume, at scale. Principal engineer Andy Swing, who ended up leaving Google since our visit, was there for the first launch. There's actually four chips inside there. It's connected to. Actually, two of those are connected to a host machine that has CPUs in it. And then all these colorful cables are actually linking together all of the Trillium chips to work as one large supercomputer. Google data centers still rely heavily on chip giants like Intel and AMD for Central Processing Units (CPUs) and Nvidia for Graphics Processing Units (GPUs). Google makes a different category of chips called ASICs (application-specific integrated circuits), which are more efficient because they're built for a single purpose.

Google's best known for its AI-focused ASIC, the TPU. But it also makes ASICs to power YouTube, called VCUs, Video Coding Units. And just like Apple, Google also makes custom chips for its devices. The G4 powers the new, fully AI-enabled Pixel 9, and the new A1 powers Pixel Buds Pro 2. But the TPU is what set Google apart because when it launched in 2015, it was the first of its kind. So the AI cloud era has completely reordered the way companies are seen. And this silicon differentiation, the TPU itself may be one of the biggest reasons that Google went from the third cloud to being seen truly on parity, and in some eyes, maybe even ahead of the other two clouds for its AI prowess. Amazon Web Services announced its first cloud AI chip, Inferentia, in 2018, three years after Google's came out. Microsoft's first custom AI chip, Maya, wasn't announced until the end of 2023.

In order to stay differentiated, to stay competitive, to stay ahead of the market, and to not become overly dependent on any supply chain, partner, or provider, they needed to do more, build more in-house. According to Newman's team's research, Google TPUs dominate among custom cloud AI chips, with 58% of the market share, and Amazon comes in second at 21%. In 2017, a group of eight Google researchers wrote the now-famous paper that invented the transformer, the underpinnings of today's generative AI craze. The invention, Vahdat says, was made possible by TPUs. The transformer computation is expensive, and if we were living in a world where it had to run on general-purpose compute, maybe we wouldn't have imagined it. Maybe no one would have imagined it. But it was really the availability of TPUs that allowed us to think, not only could we design algorithms like this, but we could run them efficiently at scale.

Still, Google has faced criticism for some botched product releases in the current rat race of generative AI, and its chatbot, Gemini, came out more than a year after OpenAI's ChatGPT. Dozens and dozens of customers are leveraging Gemini every day, including some of the most familiar names out there, whether it's Deutsche Bank, Estée Lauder, and many, many others that are household names, McDonald's, if you like, and others. Was Gemini trained on TPUs? Gemini was trained and is served externally entirely on TPUs. Back in 2018, Google expanded the focus of TPUs from inference to training AI models. Version two was actually a pod that connected 256 TPUs together. Now version five is in production, which connects almost 9000 chips together. The real magic of this TPU system is that you actually can interconnect everything over fiber optics dynamically. And so you can build small or as large of a system as you want.

With version two in 2018, Google also made its TPUs available to third parties alongside market-leading chips like Nvidia's GPUs, which are still used by most cloud customers. If you're using GPUs, they're more programmable, they're more flexible, but they've been in tight supply. The AI boom has sent Nvidia's stock through the roof, catapulting the chipmaker to a $3 trillion market cap in June, surpassing Google's parent company Alphabet, and jockeying with Apple and Microsoft for position as the world's most valuable public company. Being candid, these specialty AI accelerators aren't nearly as flexible or as powerful as Nvidia's platform, and that is what the market is also waiting to see is can anyone play in that space? Now that we know Apple's using Google's TPUs to train its AI, the real test will come as it rolls out those full AI capabilities on iPhones and Macs next year.

They were renting chips from Google for about two bucks an hour, times a gazillion chips to train their AI models. So they didn't even need Nvidia. All the market pull is coming from Nvidia, but longer-term, people are just going to want to do AI things. And when they want to just do AI things, they may be just as happy to do it on a TPU or do it on another homegrown piece of AI dedicated silicon. But developing alternatives to Nvidia's hugely powerful and expensive chips is no small feat. It's expensive. You need a lot of scale. And so it's not something that everybody can do. But these hyperscalers, they've got the scale and the money and the resources to go down that path. But the process is so complex and costly that even the Googles of the world can't do it alone. Since the very first TPU ten years ago, Google's partnered with Broadcom, a chip developer that also

helps Meta design its AI chips. Broadcom says it's spent more than $3 billion on R&D to make these partnerships happen. AI chips, they're very complex. There's lots of things on there. So Google brings the compute. Broadcom does all the peripheral stuff. They do like the I/O and the SerDes and all of the different pieces that go around that compute. They also do the packaging. Then the final design is sent off to be manufactured at a fabrication plant or fab, primarily those owned by the world's largest chip maker, Taiwan Semiconductor Manufacturing Company, which makes some 92% of the world's most advanced semiconductors. Do you have any safeguards in place should the worst happen in the geopolitical sphere between China and Taiwan? Yeah, it's an important question. And it's certainly something that we prepare for and we think about as well. But we're hopeful that actually it's not something that we're going to have to trigger.

I think the entire world is at the same risk. It's not unique to Google. It's not unique to Amazon. It's not unique to Apple. It's not unique to Nvidia. If Taiwan is not given the appropriate support, if it deals with unexpected end-of-day circumstances, it is not only going to set back any one of these companies; it's going to set back the whole world. That's why the White House is handing out $52 billion in CHIPS Act funding to companies building fabs in the US, with the biggest portions going to Intel, TSMC, and Samsung, so far. Intel and TSMC are putting a lot of their own money into this as well. I'm heartened to see that. But I mean, it's going to take a long time to duplicate. So let's hope that it doesn't need to be duplicated. Risk aside, Google just made another big chip move, announcing its first general-purpose CPU.

Axion will be available by the end of the year. Now we're able to bring in that last piece of the puzzle, the CPU. And so a lot of our internal services, whether it's BigQuery, whether it's Spanner, YouTube, advertising, and more, are running on Axion. But Google is late to the CPU game. Amazon launched its processor Graviton in 2018. Alibaba launched its own server chip in 2021, and Microsoft announced its CPU in November. Why didn't you do it sooner? Our focus has been on where we can deliver the most value for our customers, and there it has been, starting with the TPU, our video coding units, our networking. We really thought that the time was now, starting a couple of years ago. Again, these things are a number of years in the making to really bring our expertise to bear on the ARM CPUs. I don't fault Google for pacing out the launch of Axion in a more

delayed fashion. This wasn't as differentiated. It's not as differentiated. To me, it is more of a supply game. It's more of a margin and vertical integration game for the company. Whereas the TPU was truly differentiated. Six generations, ten years of experience. All these processors from non-chip makers, including Google's, are made possible by Arm Chip architecture, a more customizable, power-efficient alternative that's been gaining traction over the traditional x86 model from Intel and AMD. Power efficiency is crucial because, by 2027, AI servers are projected to use up as much power every year as a small country. With TPUs, the ability to customize greatly boosts power efficiency. This is our second-generation optical circuit switch. So, our large TPU supercomputers are actually optically interconnected. It allows us to dynamically link together collections of TPU chips to custom-tailor the dimensions to the job that's running. This is developed all in-house by us.

Power is a huge thing now, and you know anything you can do to try to improve efficiency, lower costs, and save power, I think you're going to do. Google's latest environmental report showed emissions rose nearly 50% from 2019 to 2023, partly due to data center growth for powering AI. Without having the efficiency of these chips, the numbers could have wound up in a very different place. We remain committed to actually driving these numbers in terms of carbon emissions from our infrastructure 24/7, driving it towards zero. Training and running AI also takes a massive amount of water to keep the servers cool so they can run 24/7. That's why with the third generation of TPU, Google started using direct to chip cooling, a new way to cool servers that uses far less water, and that's also being used by Nvidia's latest Blackwell GPUs. We have four chips, and these are our liquid cooling lines that come in.

There's essentially a cold plate here that has little fins in it, and it picks up the heat from the chip, puts it into the water, and that comes back out. Despite challenges from geopolitics to power and water, Google is committed not only to its generative AI tools but to making its own chips to handle the massive compute required by the craze. I've never seen anything like this, and no sign of it slowing down quite yet. I think it's fair to say that we really can't predict what's going to be coming as an industry in the next five years, and hardware is going to play a really important part there.

How Google Makes Custom Cloud Chips That Power Apple AI And Gemini

Welcome to the official YouTube channel for CNBC. From 'Wall Street' to 'Main Street' to award winning original documentaries and Reality TV series, CNBC has you covered. Experience special sneak peeks of your favorite shows, exclusive video and more.

For real-time news updates from around the world, visit: http://www.cnbc.com

For CNBC's primetime TV lineup, visit http://tv.cnbc.com

For CNBC's International YouTube channel, visit: http://www.youtube.com/cnbcinternational

CNBC

Google's Silicon Valley lab is testing its own Tensor Processing Units (TPUs), which power the company's services like search and YouTube.

TPUs are also used to train AI models, including Google's chatbot Gemini and, recently, Apple's AI.

Google's push for custom AI chips began almost ten years ago, and the company has since influenced other tech giants like Amazon and Microsoft to follow suit.

Google was the first to launch a large-scale hardware accelerator for AI, with its TPUs (application-specific integrated circuits) setting the company apart in the cloud AI market.

The TPUs' efficiency and availability played a crucial role in the invention of the transformer model, which underpins today's generative AI.

Google's in-house chip production has allowed the company to dominate the custom cloud AI chip market with a 58% share.

Google's Gemini and TPUs: A Game Changer in AI

Google's Gemini chatbot, launched over a year after ChatGPT, is now used by major companies like Deutsche Bank and McDonald's.

The chatbot is trained and served on TPUs, with Google's TPU system evolving from 256 to nearly 9000 chips.

Apple's use of Google's TPUs for AI training raises the stakes, as the market seeks alternatives to Nvidia's dominant and expensive chips.

Google partners with Broadcom for AI chip design, with Broadcom investing over $3 billion in R&D.

The complex chip-making process relies on Taiwan Semiconductor Manufacturing Company, which produces 92% of the world's advanced semiconductors.

Google's late entry into the CPU market with Axion, its first general-purpose CPU, aims to enhance internal services, but the company has faced competition from Amazon, Alibaba, and Microsoft in launching their own processors.

Google's Impactful Chip Architecture and AI Power Efficiency

Google's chip architecture, particularly the Arm Chip, is gaining traction for its customization and power efficiency, crucial for the growing demand of AI servers.

The company's TPU supercomputers use optical interconnection to dynamically link chips, boosting power efficiency and reducing costs.

Google's commitment to driving down carbon emissions and using direct chip cooling for servers demonstrates its dedication to sustainability while handling the massive compute required for AI.