OpenAI Dev Day: Key Announcements and Company Changes

OpenAI's recent Dev Day unveiled groundbreaking APIs for voice and vision, empowering developers with advanced AI capabilities.

### OpenAI Dev Day Highlights and Industry Updates
* 🎉 OpenAI Dev Day recently showcased significant announcements, including a new **Realtime API for advanced voice capabilities**, enabling developers to create **fast speech-to-speech applications**.
* 🎤 The Realtime API supports natural conversations with **six preset voices** and introduces audio input and output in the chat completions API, with pricing set at **$5 per million text input tokens** and **$100 for audio input tokens**.
* 🖼 Developers can now fine-tune Vision models through the API, enhancing image understanding for applications like **autonomous vehicles and medical image analysis**, with initial fine-tuning available for **free until August 2024**.
* 💾 OpenAI has introduced prompt caching, allowing developers to **reuse input tokens**, leading to a **50% discount on costs** and **improved processing speeds**, although cached responses are cleared after **5 to 10 minutes of inactivity**.
* 🔍 A new model distillation feature allows developers to create **cost-efficient models** using outputs from larger models, enhancing performance while **reducing latency and costs**.
* 🚀 OpenAI is transitioning from a nonprofit to a for-profit benefit corporation, with reports suggesting **significant equity for CEO Sam Altman** amidst ongoing discussions about the **complexities of this shift**.
* 🏛 In regulatory news, California's governor vetoed a proposed AI bill deemed **too broad and premature**, with industry leaders advocating for a **more measured approach to AI regulation**.
* 🆕 New models, Liquid Foundation Models, have been introduced, boasting **excellent performance metrics** and positioning themselves as **state-of-the-art** in their respective categories, although they face challenges in **zero-shot coding tasks** and **precise numerical calculations**.

The big news of this week is OpenAI Dev Day just happened, and they made a bunch of awesome announcements that we're going to go over first. Plus, we have more stories from OpenAI and so much more. Let's dive into it all right. So first, OpenAI Dev Day just happened, and they first of all have a really cool website with a really cool animation. Very simple. So here are the announcements you need to know about. First, they announced an API for essentially advanced voice mode. It's called Realtime API. Developers can now build fast speech-to-speech experiences into their applications, similar to ChatGPT's advanced voice mode. The Realtime API supports natural speech-to-speech conversations using the six preset voices already supported in the API. So if you want to build amazing voice support into your applications, now you can. Plus, they're also introducing audio input and output in the chat completions API for low-latency use cases. The Realtime API uses both text tokens and audio tokens. Text input tokens are priced at $5 per million and $20 per million for output tokens. Audio

$100 per million input tokens and $200 per million output tokens. This equates to approximately 6 cents per minute of audio input and 24 cents per minute of audio output. That is incredible hyper-intelligent real-time voice for just pennies per minute. Absolutely incredible world we're living in. Next, you can now fine-tune Vision models through the API. Developers can customize the model to have stronger image understanding capabilities, which enables applications like enhanced visual search functionality, improved object detection for autonomous vehicles or smart cities, and more accurate medical image analysis. So, here's an example of a speed limit sign that apparently was incorrectly tagged by the base GPT-4 Vision model but was correctly tagged by the fine-tune version. And the massive company Grab, which is kind of like Uber in the United States, is using this model for image detection and understanding on the road. They have some other awesome use case examples which you could check out. I'll drop the link in the description below. And in terms of pricing, what they say here is we're

offering 1 million trading tokens per day for free through August 31st of 2024 to fine-tune GPT-40 with images. Then, after that, it'll cost $25 per million tokens, and inference will cost $375 per million input tokens and $15 per million output tokens. So, definitely check out fine-tuning of vision. Next, one of the most important features of artificial intelligence, in my opinion, in the future is caching. We don't necessarily need to run every single prompt through the LLM every single time, especially if you're giving the same prompt over and over again. And it turns out, at scale, that happens often. So, now you can cache. Let's read a little bit about it. Many developers use the same context repeatedly across multiple API calls when building AI applications, like when making edits to a codebase or having long multi-turn conversations with a chatbot. Now, they're introducing prompt caching, allowing developers to reduce cost and latency by reusing recently seen input tokens. Developers can get a

50% discount and faster prompt processing times. Now, I've talked about this quite a bit. Caching is incredibly important for those two reasons: latency and cost decreasing. And as you can see here, caching is across the board on all of their current models. So, here's the uncashed tokens right there and then the cached version. Now, here's the thing: if you're going to cache it, why even pay OpenAI to do it at all? I mean, obviously, it's much easier to just let them handle the caching. Caching is actually a very hard problem to solve, which is something that I dealt with many times at my previous company. Caching is just difficult. So, if you're just looking to do things simply, you can just let OpenAI take care of it. But for essentially free, you can cache things yourself. And I believe that there's going to be a lot more open-source tooling that allows you to do caching locally. Now, something interesting I noticed at the bottom here: caches are typically cleared after 5 to 10 minutes of inactivity and are always

removed within 1 hour of a Cash's last use, so that is their cash timing, which seems aggressive to me, because if I give a prompt today and then I give the same prompt tomorrow, I probably want the same exact response. But every use case is different, so you tell me what you would want to see with caching. Next, we have big models fine-tuning little models. Now OpenAI is offering model distillation in the API. Fine-tune a cost-efficient model with the outputs of a larger Frontier Model, all on the OpenAI platform. So, so basically kind of like synthetic data. You use this larger model to create a bunch of data that is specific to your use case, specific to your business, train a smaller model, and then get lower latency and lower cost because of that. So, a lot of these updates that OpenAI announced today are specifically around giving developers the most options possible, which I love. Model distillation includes three core concepts: stored completion, basically the input-output pair, so you can use that data to train other models, EV vales, so

you can run custom valuations against your fine-tuned model to make sure it's performing up to your standards, and fine-tuning, of course, which is exactly what it sounds like, taking that data and fine-tuning your model. So, those are the major updates from OpenAI Dev Day today. At the last moment, I was fortunate enough to be invited by OpenAI, but I couldn't make it this time. Definitely the next time they invite me to something, I will be there. So, congrats to OpenAI for these incredible developer-focused product announcements. Next, speaking of OpenAI, the thing that we all knew was going to happen eventually, they're removing the nonprofit status essentially. So, in an exclusive by Reuters, OpenAI to remove nonprofit control and give Sam Altman equity. Now, it was reported that Sam Altman was going to get a lot of equity, something like $7 billion worth or $10 billion worth, an absurd amount. And here's the thing, Sam Altman less than a year ago testified in front of the Senate saying, I make no, I paid enough.

for health insurance. I have no equity in OpenAI, and now he's talking about getting paid. But here's the other thing: he also denied that he was going to get this massive payday, and that was only a couple of days ago that that happened. So who knows what the actual truth is right now? We're just going to have to wait and find out. I do believe that OpenAI is going to be moving away from the nonprofit status. That has been an overly complex structure that really no company should take on, and I've already made multiple videos about it, so I'm not going to rehash everything about Elon Musk and his donations and them taking donations, not paying taxes, and then converting over to a for-profit company. But all that aside, they are looking for the cleanest path forward to convert to a for-profit. According to the article, CHPT Maker OpenAI is working on a plan to restructure its core business into a for-profit benefit corporation that will no longer be controlled by its nonprofit board. So what does a for-profit benefit

Corporation actually means in terms of the legal and tax aspects of the company. It's essentially a for-profit company, but the benefit piece of it basically means they have an external stakeholder that is in the charter of the company, and they not only have a fiduciary duty to the shareholders, but they also have a fiduciary duty to the external stakeholder. One of the most famous examples is Patagonia, and I believe their external stakeholder is the climate. I believe that's what it is. So basically, the board has a fiduciary duty not only to the shareholders but also to the climate and nature in general. So it's definitely interesting. I don't know how much it actually means, but it's in the actual board W's Mission. So yeah, they are beholden to whatever that third-party stakeholder is. It'll definitely be interesting to see how all the chips fall here because it has been a lot of complexity and a lot of friction seeing how open AI is going to transition from this essentially non-profit laboratory.

into a for-profit commercial entity. I want to thank the sponsor of this video, weights and biases. I'm so excited to be partnering with them. They've partnered a long time with one of my favorite YouTube channels, two-minute papers, and now I get to tell you about them. If you're building with all these LLMs I'm telling you about on production, you need a robust way to trace everything. Weights and biases has a new product called Weave, which is a lightweight AI developer toolkit that lets you simplify your LLM app development with just two lines of code. You can trace input metadata and output. Weave helps you run rigorous evaluations and securely manage all of your data sets and system configurations so you can focus on just building. Increase visibility and debugging better and faster evaluation. Clear visibility into cost and results and one of my favorite things, conversion of your prompts and models. Weave seamlessly integrates into your favorite APIs and libraries, including OpenAI, Anthropic, Mistral, Cohere, LangChain, LlamaIndex, and more. So if you want a

dead simple way to accelerate your AI app development. Weave is a great solution, so make real progress with your LLM development. Visit Weights and Biases, check out their Weave product. I'll drop a link to it in the description below wb.slmb. Be sure to check it out, and now back to the video. Next, continuing on OpenAI, Mir Moradi and other leaders have left the company all of a sudden last week on Wednesday the 25th. Mira Moradi has left OpenAI. Mira Moradi was the CTO. She was pivotal for many years growing this company into what it is. She was also part of the Mutiny, I believe, reportedly when they tried to oust Sam Altman. Now, from this very famous picture of the four OpenAI founders, I believe there's actually five, but there's only four listed here. There's only one left who is actively working at OpenAI, and that is Mr. Sam Altman himself. Ilas Husky left to start SSI. Mir Moradi just left. We'll see what she does. Greg Brockman is

on an extended leave of absence, although he is planning on coming back. But here's the interesting thing: Mea gave him no heads up that she was leaving. When Merera informed me this morning that she was leaving, I was saddened but, of course, supported her decision. That is not normal for your CTO to just suddenly leave. And he also shared that Bob and Barrett have also decided to depart from OpenAI. Now, those are also two high-level executives, so there is a lot of talent drain going on. I'm not too worried for OpenAI; they are still an incredible company with lots of talent. Sam Altman is an incredible executor, and of course, this is going to affect them, but I don't think it's going to really change the trajectory of this company. And everybody who left basically wrote very boilerplate posts on X, talking about how wonderful the experience was and how they're leaving OpenAI in such a good place. But you kind of have to ask yourself: if you're on the track to achieve AGI, why

would you leave if you're making a ton of money, doing amazing research, having fun building incredible products that the entire world is using, and really changing the world? Why would you leave? And that is the only part that I can't really answer. So let me know what you think in the comments. Next, in incredibly great news, California's governor Gavin Newsom has vetoed the new AI bill that was almost set to be law. And I live in California, so I'm especially glad to see this. But at the last moment, he vetoed it. And here's a few things to note: a lot of incredibly important people in the world of AI basically said this bill is way too early, way too broad, and doesn't have well-defined boundaries. And this would especially harm open-source companies. And there were a ton of AI luminaries and AI researchers, really important people in the world of AI, much more knowledgeable about AI than any of the politicians in California. And here are just a few: Jason Quan, Chief Strategy Officer at OpenAI; I know Andrew Ng.

was pretty pivotal in getting this vetoed. Yan Laon, of course, Mark Andreessen from A16Z, and just so many more. But it wasn't all opposition. Elon Musk even said that this bill probably has more pros than cons, and Anthropic, as a company, stated that this bill's benefits likely outweigh its risks. But it has been vetoed, and here's my take on it: I think it's too early to regulate artificial intelligence. If I think back to the early days of the internet, if they would have just regulated it and just strangled innovation, the internet would not be what it is today. And we've already seen what happens when you over-regulate or regulate things too early, like in the EU. Basically, all these AI companies are just not releasing products there, and there's not really a lot of AI companies being started there. And not only that, there's already active discussions at the federal level, the US federal level, about regulating AI. And so why have a separate bill for California this early? It just didn't make any sense.

I was against it. I'm glad to see they vetoed it. Good on California. Next, P Labs just released Pika 1.5, and it is absolutely stunning. Look at some of these videos. These were all created with P 1.5. I mean, a lot of these could easily just become Pixar movies, and nobody would be any of the wiser. According to Pika's post, more realistic movement, big screen shots, and mind-blowing Pika effects that break the laws of physics. There's more to love about Pika than ever before. Next, there's been some drama in the world of Open Source AI code mode editors. This is a weird story, but we're going to cover it. So, there's this company Pair AI, which is backed by YC. They basically forked a very well-known open-source product, but they did something weird with the licenses, and then they released it. They actually got community noted during their announcement. It was a whole drama that a lot of people weighed in on, and here's one of the founders, Frying Pan. Let's see.

what he has to say. We were excited to announce our product and company on Twitter yesterday, but you've all helped us understand that we did so in a way that was haphazard at best and at worst offensive to the open-source community that we love. We made two big mistakes. First, we kicked off our project by forking Continue and VS Code. We did this because we want to build an open-source AI code editor with the form factor of VS Code and with all the usefulness of Continue's extension. This double fork was the fastest way to get something up and running for us to experiment and build off of. We tried to be transparent about what we'd done as much as possible since the beginning of our journey, but that wasn't good enough. Second, we flubbed the license. Our intention was to use the Apache 2.0 license like Continue does. Our repo has two modules, and we got this right in one of them and wrong in the other. We thought the license in the root repo wasn't that important, so we just

generated one that we thought was open. In hindsight, this sort of haphazard treatment was not only technically incorrect but offensive to everyone who contributes to open source projects. And also, what seemed to rub people the wrong way was his announcement because it said, I just quit my $270,000-a-year job to do this. And I get it, it's very click-baity, and he was just trying to say, Look how important this is to me, so I get it. But I also get why that would rub people the wrong way. Now, the fact that he basically just came out, owned it, explained what he's going to do going forward, I appreciate that. I always am fine giving people second chances. And continue also posted about it, so continue Dev, which by the way, fantastic AI coding product if you ever want to use it. It is a fork of VSS code, I believe that's the right way to say it. And they basically said, Pair did things the wrong way. Here's what happened, and we appreciate that they are owning it.

and have acknowledged it and will do better. Our view is that Pari should start over from scratch, make a true fork of the continue repository rather than a copy, retain the Apache 2.0 license at all times, reapply whatever changes they wish to make to continuous code in compliance with the license, and the founder right there, Nang from PA AI. Thank you, we will do exactly this, and sorry again. So, it seems all good worked out. Pair AI made a mistake, and they owned it, and they're going to fix it. Can't really ask for anything more than that. But speaking of making mistakes and fixing it, where has Matt Schumer been? We're still waiting on what happened with the reflection 70b model. The last thing we heard was over 3 weeks ago, and we really haven't heard anything else. I was really hoping to hear more and get a little bit of insight into what happened. Maybe they're retraining the model, but to go completely silent is just such a bad look. At least give us information at

least least communicate that is the least you can do. And for our last story, we have new models liquid AI. Today, we introduce liquid foundation models to the world with the first series of our language LFMs, liquid foundation models. I don't know if we need another acronym, but here we go: 1B, 3B, and 40B. And the performance is excellent. The 40B is a mixture of experts model with 12 billion active parameters, so a large model but very efficient. And it outperforms QU2, Mixol 47B, Jamba 1.5, which honestly isn't that hard given I tested it and it was pretty bad. But all three of the family of models seem to really dominate every other model in its class. So, I plan on testing it. What do you think? Should I test it? Let me know in the comments. LFM 1B performs well on public benchmarks in the 1B category, making it the new state-of-the-art model at this size. This is the first time a non-GPT

architecture significantly outperforms Transformer-based models. So, the important thing to note is it is not Transformer-based. The only experience I have with non-Transformer-based models is the Mamba model, and again, it didn't perform well. So, I have my hopes up, but I'm also not going to hold my breath. The 3B model delivers incredible performance for its size. It positions itself as a first place among 3B parameter Transformers, hybrids, and RNN models. And then the 40B offers a new balance between model size and output quality. And as we can see here, compared to all of these other models on the list, it performs very well. Well, so one thing I really like that Liquid did is actually say, Here's what it's good at, here's what it's not good at. So, let's look at what language LLMs are good at today: General and expert knowledge, mathematics and logical reasoning, efficient and effective long context tasks, primary language of English, but it also supports other languages. And what it's not good at: zero-shot coding tasks. Okay, we'll

definitely test that precise numerical calculations. I think that's 9.11 versus 9.9, maybe time-sensitive information. Obviously, it doesn't have up-to-date information unless it's able to crawl the web. Counting RS in the word strawberry. Okay, noted. I will definitely test that, and human preference optimization techniques have not yet been applied to our models. So, you can get more information about the models right here on their blog post. I'll drop links in the description below, of course. So, that's it for today. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one. Bye.

OpenAI Dev Day, CTO Suddenly Departs, AI Bill Vetoed, Pika 1.5, and New Models!

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

OpenAI Dev Day made several significant announcements, including the launch of a Realtime API for advanced voice mode, allowing developers to build fast speech-to-speech experiences.

The API offers audio input and output for chat completions, with competitive pricing for tokens.

Additionally, OpenAI is now offering fine-tuning for Vision models through the API, providing developers with the ability to customize image understanding capabilities.

The introduction of prompt caching aims to reduce cost and latency for developers by reusing input tokens across multiple API calls.

OpenAI is implementing aggressive cash timing for model use and offering model distillation in the API to create cost-efficient models.

The company is also removing its nonprofit status and restructuring into a for-profit benefit corporation, giving Sam Altman a significant equity stake.

This transition aims to simplify the company's structure while maintaining a fiduciary duty to both shareholders and an external stakeholder.

OpenAI has experienced a sudden exodus of key leaders, including CTO Mira Moradi, who played a pivotal role in the company's growth.

Sam Altman is now the only founder actively working at OpenAI, while other high-level executives, including Ilas Husky and Greg Brockman, have also departed or taken leaves of absence.

Despite the talent drain, the speaker believes OpenAI will continue to thrive under Sam Altman's leadership.

California's Governor Gavin Newsom vetoed a controversial AI bill, with many AI experts arguing it was too early and broad.

The veto was supported by notable figures in the AI industry, while some, like Elon Musk, saw potential benefits in the bill.

5, showcasing stunning new features and realistic movement, while drama unfolded in the open-source AI community over a company's controversial fork of a well-known product.

Pair AI's announcement on Twitter sparked backlash for their haphazard approach to forking Continue and VS Code, as well as their incorrect handling of the Apache 2.

The founder acknowledged the mistakes and committed to starting over, making a true fork of the Continue repository and retaining the correct license.

Meanwhile, there is still uncertainty surrounding Matt Schumer's silence on the Reflection 70B model, leaving many waiting for updates.

Liquid AI has launched its new series of Liquid Foundation Models (LFMs) with 1B, 3B, and 40B parameters, showcasing excellent performance.

The 40B model is a mixture of experts with 12 billion active parameters, efficiently outperforming other models in its class.

The LFMs are the first non-Transformer-based models to significantly outperform Transformer models, with the 1B model setting a new state-of-the-art benchmark.