AI Chip Advancements and Industry Dominance

Soou, a new AI chip, claims to run 500,000 tokens per second, outperforming GPUs, signaling a shift in AI hardware development.

### AI Chip Startup Etched's SOHU Challenges Nvidia's Dominance; OpenAI Delays Advanced Voice Mode Rollout
* 🪙  Etched introduces **SOHU, a specialized AI chip for Transformer models**, claiming to be **over 10 times faster and cheaper** than Nvidia's Next Generation Blackwell GPUs.
* 📉  GPUs are facing challenges in performance improvement due to increasing size, leading to the need for **specialization**, similar to Bitcoin mining in 2014.
* 🔒  Etched believes in the "**hardware lottery**," where the **architecture that performs best on hardware wins**.
* 🔊  OpenAI delays the rollout of the advanced voice mode of ChatGPT, aiming to start the alpha with a **small group of ChatGPT Plus users** in the fall.
* 🤖  The advanced voice mode enables AI to understand and respond with **emotions and non-verbal cues**.
* 📊  Hugging Face CEO Clément Delangue introduces a new Open LLM Leaderboard, revealing that **Quen 72b is the top performer** and highlighting the **dominance of Chinese open models**.
* 🚀  CLA 3.5 Sonnet by Anthropic secures the **number one spot in the coding arena** and the **number two spot overall**, surpassing Opus at **five times lower cost** and competing with **GPT-3.5 and Gemini 1.5 Pro models**.
* 🍏  Apple reportedly considers integrating Meta AI's LLM 3 models into Siri via an API call but later **shelves the idea over privacy concerns**.

again, it's just been a few days since my last news video, and so much has happened. We're going to be talking about AI chips, OpenAI, Apple Cloud 3.5s dominance, and what looks to be the future of video games. Let's get into it. The first story today is about a new AI chip company called etched, and it claims to be able to generate over 500,000 tokens per second running llama 70b. So this chip is called sohu, I believe, and it lets you build products that are impossible on GPUs. One server with eight of these replaces 160 Nvidia h100s, and this is all assuming this is true because I don't think this is actually in production yet. Now here's how it's able to do that. Soou is the first specialized chip as6 for Transformer models. Now, if you remember, in the earlier days of crypto mining, specific chips were created for Bitcoin mining, and these chips were especially made just to crack that algorithm. And this chip is specific to the Transformer model, and they even

say that we get way more performance so Hu can't run CNN's convolutional neural networks LSTMs SSMS or any other AI model every major AI product chbt CLA Gemini Sora is powered by Transformers within a few years every large AI model will run on custom chips it is more than 10 times faster and cheaper than even Nvidia's Next Generation Blackwell GPUs one so hu server runs over 500,000 llama 70b tokens per second 20 times more than an h100 23,000 tokens per second and 10 times more than a B2 45,000 tokens per second and here are the benchmarks now interestingly enough they do not compare themselves to Gro which I wonder why and it says here GPUs aren't getting better they're just getting bigger in the past four years compute density has only improved by 15% next-gen GPUs are now counting two chips as one card to double their performance with Mo's loss slowing the only way to improve

performance is specialization, and they specifically call out Bitcoin mining here. So when Bitcoin miners hit the market in 2014, it became cheaper to throw out GPUs than to use them to mine Bitcoin. And that's exactly what happened. Everybody started transitioning to ASICs. They go on to say, Transformers have a huge moat. We believe in the hardware lottery. The architecture that wins is the one that runs fastest and cheapest on hardware. I'm not really sure what they meant by hardware lottery in this context. Typically, the hardware lottery means when you get a GPU, some of them are just made better than others. AI labs to spend hundreds of millions of dollars optimizing kernels for Transformers. Startups use special Transformer software libraries like TRT, LLM, and VM, which offers features built on Transformers like speculative decoding and tree search. Once SOOU hit the market, we will reach the point of no return. Transformer killers will need to run faster on GPUs than Transformers run on SOHU. If that happens, we'll build an ASIC for that tool. And on their website.

scale is all you need for super intelligence. So, if you want to learn more about Soou and the technical details, check out the website et.com. I'll drop a link to that in the description below. Next, it seemed like the OpenAI voice capabilities were taking forever to come out, and now we know why. They're actually delaying them. And as a reminder, the GPT-40 voice capabilities were essentially that movie Her. Hey chat, GPT, how are you doing? I'm doing fantastic, thanks for asking. How about you? All right, so we're sharing an update on the advanced voice mode we demoed during the spring update, which we remain very excited about. We plan to start rolling this out in Alpha to a small group of ChatGPT Plus users in late June, but need one more month to reach our bar for launch. For example, we're improving the model's ability to detect and refuse certain content. This is disappointing. This is very similar to what it felt like when they announced and demoed Soou, and then we basically have no idea when that's actually coming.

and I'm guessing they were able to jailbreak it. We're also working on improving the user experience and preparing our infrastructure to scale to millions while maintaining real-time responses. As part of our iterative deployment strategy, we'll start the alpha with a small group of users to gather feedback and expand based on what we learned. We are planning for all Plus users to have access in the fall, so that's great, right around the corner. Exact timelines depend on meeting our high safety and reliability bar. We are also working on rolling out new video and screen sharing capabilities. We demoed separately and will keep you posted on that timeline. Chat GPT's advanced voice mode can understand and respond with emotions and non-verbal cues, moving us closer to real-time natural conversations with AI. Great and Billow wall responds with not sure what the point of flexing a live demo was if the intention was to delay the launch like this. Clearly the product wasn't ready. I could probably say the same thing about Microsoft's recall, which was

recalled both of these things. I was incredibly excited about, but now we have to wait. Next Clen, the CEO and co-founder of Hugging Face, has now announced a new Open LLM Leaderboard. We burned 300 H100s to rerun new evaluations like MML U Pro for all major Open LLMs. Some learnings: Interestingly, Quen 72b is the king, and Chinese open models are dominating overall. You all know that I tested Quen 72b, and yes, it did perform very well. I would like to see a coding-specific flavor of Quen 72b. Previous evaluations have become too easy for recent models. That is something that I found, at least with my own LLM rubric. And I asked you for suggestions for new tests in my previous video. Keep them coming. Drop it in the comments below if you have suggestions for new tests that I should use going forward on these open-source models. There are indications that AI builders have started to focus on the main evaluations too much at the expense

of model performances on other ones. Now, interestingly, a lot of you have mentioned and suggested that some of these model makers might actually use my tests to train their models. I didn't really think that was true because I'm not actually doing a formal benchmark, and it's just a little YouTube channel. But it seems like open-source model makers are focusing on these major public benchmarks, and bigger is not always smarter. So that is something that I've been thinking a lot about lately. Think about this: the Llama 3 88b model is substantially better than the Llama 2 7B model, so just slightly bigger but much better performance. And they've decided to cover the following general tasks: knowledge testing, reasoning on short and long context, complex math abilities, and tasks well correlated with human preference like instruction following. And to do that, they have these benchmarks: MLU, ProGPT, QAMSR, which I've never heard of, multip soft reasoning, very cool math, if eval, and BBH. And here are the results. As you can see here, quen

72b instruct is number one metal llama. 370b instruct number two. 53 medium 4K instruct number three ye 1.5 34b number four and so on. So if you want to learn more, if you want to read about the details, if you want to see the benchmarks themselves, I'll drop a link to hugging face and this specific page in the description below. And speaking about benchmarks and leaderboards by lm.org CLA 3.5 SAA has just made a huge leap securing the number one spot in coding arita hard prompts Arena and number two in the overall leaderboard. The new Sonet has surpassed Opus at five times lower cost in competitive with with Frontier models GPT 40 and Gemini 1.5 Pro across the board. Now in a previous video, I tested claw 3.5 Sonic and yes it is the best model I've ever tested. In fact, that was the video in which I asked for new tests because it completely demolished my tests. So huge congrats to anthropic and now it is

number one in coding, number one in hard prompts, and number two overall. And remember, this is the Sonnet model. This is 3.5 Sonet Opus. Their largest model is still coming, 3.5 Opus. That is so very cool to see. I'm now paying for Claude, and that is my go-to model. And Anton OA has also showed Claude 3.5's performance versus GPT-40 on coding, and this is build success. GPT-40 wins, Claude 3.5 wins on task success, and full project success. Let's see what else he says about it. Code that compiles fails a bit more for Claude, small difference, and pass human QA fails significantly less for Claude. And even more interesting is the realistic benchmarks, qualitative results. So he says Claude more verbose, nice for long pieces of code, makes generation slower, which is interesting because I guess the overall generation, because the code is longer, it might make sense, but from what I've noticed, the actual tokens per second seem to be faster with Claude, and generally not.

desirable in an agent setting which is interesting and I haven't tested. If you want to see me test that, let me know in the comments. Does not follow instructions in large prompts as reliably as GPT 40. It tends to miss crucial instructions. For example, how to format output. We experimentally switched to Claud for a right long code sub-agent at lovable. Unfortunately, we decided to revert. One of the things that I'm going to be testing going forward for all the models is its ability to format its output in specific formats and really Json. Next, Elon Musk posted an absolutely incredible video of two rockets landing for reuse at the same time. Look at this video. This is the stuff of science fiction. Imagine how much science and engineering had to go into this to get these massive rockets to actually be able to come back down to earth land for later reuse. So congratulations to SpaceX. Really cool accomplishment. Keep up the amazing work. And I see Matt Wolf commented this will

never not be impressive to me. Completely agreed. Next, Twitter user chubby, who always posts really interesting stuff, posted this video that is rendered completely using AI. This is not real engine, this is actually AI, and the quality is mind-blowing. So what we're looking at is an AI-generated video of what looks to be kind of a Call of Duty game. The sound is AI-generated, the visuals are AI-generated. I can see a little bit of morphing, but overall it looks incredible and very, very realistic. Now, the amount of compute it will take to do this in real time is going to be tremendous. And so we're not quite here yet, but as Jensen, the CEO of Nvidia, has said, this is truly the future of video games and will really take video games to the next level. And just because people were doubting that this was an actual video game created by AI, here is another clip of it. Let's look at this one. So here's another version. Again, you could see a little bit of morphing.

little bit of clipping here and there, but overall it looks really good. It again is kind of like a Call of Duty game. All right, and last in the quickest turnaround in reporting ever. Just a couple days ago, it was reported that Apple was in talks with Meta AI to integrate Llama 3 into Siri, the same way that OpenAI is integrated into Siri. So not a deep integration, just simply an API call. But just a day or two later, it is now reported that they are not considering that anymore, and specifically over privacy concerns, which is really surprising because if Apple actually took the Llama model hosted it themselves, they would be able to control the privacy. So maybe that means they were depending on Meta to actually operate the inference endpoint. So days after the Wall Street Journal reported that Apple and Meta were in talks to integrate the latter's AI models, Bloomberg's Mark Gurman said that the iPhone maker was not planning any such move, and they were in talks with multiple companies to explore integration. And I know they were also in

talks with Google, or at least it was reported as such. They shelved the idea of putting meta AI models on iPhones over privacy concerns. The report also noted that partnering with the social networking company won't do a lot of good for Apple's image, given that the Cupertino-based company has continuously criticized Meta's privacy practices. Very true, but again, I think they could have easily hosted the model themselves, them being Apple and all the privacy concerns they would have control over. So I'm not sure why they didn't do that. So that's it for today, thanks for watching. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

New INSANE AI Chip, GPT4o Voice Update, Claude 3.5 Dominates, SpaceX Double Landing, AI Video Games

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

The Impact of Etched's SOHU Chip on AI Computing

Etched has developed the SOHU chip, specialized for Transformer models, claiming to outperform Nvidia's GPUs.

The chip is designed to run over 500,000 llama 70b tokens per second, offering significant speed and cost advantages.

Etched's focus on hardware specialization for AI models could potentially reshape the future of AI computing, leading to the development of new ASICs for specific tasks.

OpenAI's Delay in Releasing Advanced Voice Mode

OpenAI is postponing the launch of the advanced voice mode for ChatGPT Plus users, citing the need to meet high safety and reliability standards.

The new release date is expected to be in the fall, with a focus on gathering feedback and expanding access gradually.

The updated mode aims to provide real-time natural conversations with AI, including emotions and non-verbal cues.

The Impact of Model Performance Benchmarks

Open-source model makers are using major public benchmarks to train their models, with the Llama 3 88b model outperforming the Llama 2 7B model.

The benchmarks for these models include MLU, ProGPT, QAMSR, multip soft reasoning, if eval, and BBH, with the Llama 3 88b model leading the way.

5 SAA has also made a significant leap to secure the top spot in coding arita hard prompts Arena, surpassing Opus at a much lower cost, and showing impressive performance against GPT-40.

Elon Musk's SpaceX Achieves Incredible Feat with Simultaneous Rocket Landings

Elon Musk's SpaceX successfully lands two rockets simultaneously, a remarkable feat of science and engineering.

The AI-generated video posted by Twitter user Chubby showcases the mind-blowing quality and realistic visuals of a game, demonstrating the future potential of AI in video game development.

The videos highlight the impressive progress in both space exploration and AI technology.

Apple's Plans for Integrating Meta AI into Siri

Apple was initially reported to be in talks with Meta AI to integrate Llama 3 into Siri, but has since backed out over privacy concerns.

The company is now exploring integration with multiple other companies, including Google.

The decision to shelve the idea with Meta AI was influenced by concerns over how it would impact Apple's image, given the ongoing criticism of Meta's privacy practices.