Google and OpenAI: Major AI Model Updates and Features

Google's Gemini 2.0 Flash model excels at rapid responses and diverse applications.

### Major AI Developments: Google’s Gemini and OpenAI’s Sora Unveiled
* 🚀 This week has been monumental for AI news, particularly with **significant announcements from Google and OpenAI**.
* 🌟 Google introduced the Gemini 2.0 foundation model, which includes the **faster and more efficient** Gemini 2.0 Flash, outperforming its predecessor, Gemini 1.5, in **almost every benchmark at double the speed**.
* 💻 Users can access Gemini for free at gemini.google.com, and the AI Studio allows for **real-time interactions**, including **voice conversations** and **webcam integration** for enhanced user experiences.
* 📊 The **Gemini 2.0 Flash model** excels in **real-time data analysis**, **chatbot applications**, and **content generation**, showcasing its **versatility and speed** in processing information.
* 📱 Google also announced Project Astra, an AI assistant for mobile devices that can **recognize and respond to the user's environment**, with plans to integrate it further with Google services and even into **smart glasses**.
* 🖥 OpenAI joined the excitement by launching **Sora Turbo**, a video generation tool, which allows users to create **short videos with detailed prompts**, although it faced **server overload upon release**.
* 🖼 Additionally, OpenAI's Canvas feature now supports **Python code execution** and is available for free, enhancing user capabilities in **coding and creative writing tasks**.
* 🤖 The week concluded with various other AI advancements, including updates from Anthropic, Midjourney, and Adobe, all contributing to the **rapid evolution of AI technologies** across multiple platforms.

Well, it's been a heck of a week for AI news, especially from Google and OpenAI. Both those companies have been shipping like crazy, and I'm going to do my best to break it all down for you right meow. Google has entered what they call the

Gemini era. Their new Gemini 2.0 foundation model was released this week, and along with it was a ton of cool features and ways that this Gemini model is being used. They started with the release of Gemini 2.0 Flash, and the Flash model is a smaller model, but it

actually performs much better than Gemini 1.5, which is a larger model trained on a lot more data. But we can see by the benchmarks here that it pretty much outperforms, in almost every single area, the much larger models that came before it, and it does it all

at twice the speed. You can actually play around with the Gemini model yourself right now for free. You head on over to gemini.google.com. Now, I am a paying member of Google, so yours will probably look a little bit different, but if you click this drop-down here, you will see 2.0

Flash experimental. Now, this is a really good model, and you'll probably notice it's pretty comparable to things like Claude and ChatGPT and things like that, but there's an even cooler way to use it, which will open up even more possibilities, and that's by going to

Google's AI Studio. If you go to AI Studio.google.com, even as a free Google user, you will see Gemini 2.0 Flash experimental over here that you can use right now. And if we take a look at the pricing, well, it's currently free to

Use and test. You can also give it some extra features like structured output, code execution, function calling, grounding, and even adjust safety settings here, essentially allowing you to make it completely uncensored. But here's where it gets really cool: if we click over on 'Stream, Real Time,' we have

some additional options here. So you can talk to Gemini with real-time voice conversation; you can turn on your webcam and Gemini can see what's going on on camera; or you can even share your screen, and Gemini will be able to help you with whatever you're seeing on your screen.

Let's go ahead and try to talk to Gemini. What are some of the best use cases of this new Gemini 2.0 flash model? The Gemini 2.0 flash model is great for various use cases. It's particularly good for tasks that need quick responses, like real-time data

analysis or generating content summaries. It is also useful in chatbot applications or in any situation requiring rapid information processing. Let's go ahead and try the show Gemini and see if it'll actually look at my camera, even though it's using my camera for the recording. Can you describe what

I'm wearing right now? Sure. You're wearing a dark-colored T-shirt and a camouflage baseball cap. You also have headphones on and a beard.

Describe the background environment for what I'm doing right now. Okay, the background appears to be a room with dark blue walls. There's a

Now, sure, I can tell you what's on your screen right now. You are on a page titled "Google AI Studio", with a "Stream Real-time" section selected. The page has options for system instructions, a run settings menu, and other tools. There is also a chat box at the bottom with a

start and stop screen sharing option and a video of your screen in the bottom right corner of the page. How many tabs do I have open in my browser right now? Based on the image, it looks like you have 10 tabs open in your browser right now. Well, it didn't quite get that right.

There's quite a bit more browsers open, but it seems to work and be able to see my screen pretty well. So, theoretically, if I am trying to work through something on my computer, or I run into a coding issue, or even if I'm playing a video

game, I can have my screen share going and have the AI sort of talk me through what I'm supposed to do next. That's kind of the idea here. Now, this is an early prototype; it's still experimental, so as we saw, sometimes it can't do things very

accurately, like count tabs, but in most of my experimentation so far, it's been pretty spot-on. It's also got some additional functionality here, like spatial understanding, where you can upload a photo and combine 2D or 3D spatial analysis with reasoning. So it's

got some sample images here. Let's go ahead and use this image and say, "Label each one with the type of food it is." And you can see it's doing its little calculations here. It created these bounding boxes around it, and if I hover over it, you've got cheese, you've got a

Muffin, you've got a strawberry, etc. It labeled all these things in the image for me. Here's another image I just uploaded from my computer. Let's try a 3D bounding box; detect the 3D bounding box of... let's do the speaker. Okay, that wasn't

the type of speaker I meant, but it did find the speakers on the stage. Let's do the person on stage, and there you go; it made a bounding box kind of where I am. It's not perfect, but you get the idea of what they're going for here. You can also

upload videos and have it analyze the videos. The videos don't even need to have audio, and it will know what's going on in the video and describe the scenes and things like that, so you can theoretically find very specific points in your video. There's also a map

explorer. A lot of these features I will probably play around with in a future video. However, the new Gemini 2.0 flash is really, really impressive. But that was just the beginning of Google's announcements this week; they also announced Project Astra, which, if you pay

attention to my Shorts feed at all, or you're following me on X or Instagram, I shared my experience with Astra because Google actually flew me out to London, put me up in a nice hotel, gave me a brand new Pixel 9 phone with Astra

pre-installed on it so that I can play around with Project Astra myself. And basically, what Project Astra does is it's designed to be like an AI assistant on your phone, like an infinitely better version of Siri that can actually see what's going on around you and

Eventually, it will connect to all your Google apps, so it should eventually be able to connect to Maps, Calendars, Gmail, and things like that. Right now, it's not connected to all that stuff; it's just got the sort of vision capabilities. It can search the web; it can answer questions.

for you, and it is really, really impressive. In fact, let's go ahead and just demo it right here on this video. So I've got the phone open here. I'm going to go ahead and open Project Astra, and now if I click Start, "Hey Astra, what am I"

looking at right now? Based on the view provided, you seem to be observing a setup with multiple screens, one of which is displaying a live feed from a smaller device, perhaps a phone or a monitor. There is also a smaller monitor mounted to the top of the larger monitors. Can you tell me what

kind of camera this is right here? Based on the logo, it appears to be a Sony camera. I'm unable to specify the model of the camera, though. Can you tell me what kind of microphone this is? This is a Shure SM7B microphone. It is a popular choice for vocal recording and

broadcasting. Can you tell me where this picture on the wall was taken? Based on the appearance of the shoreline and vegetation, it's plausible that this image was captured in a tropical location with volcanic terrain, potentially in Hawaii or a similar region. It is only a guess.

Well, you're right; it's Maui. Glad to have gotten it correct. Another thing that this is really, really good at is you can take a book, like I've got this book here in my hand, open up to any page in the book. I could briefly flash the camera at the

Page on the book, and it will know everything about that specific page. So check this out: Can you tell me what's going on on this page of this book? The page is discussing negotiation strategies, specifically focusing on the topic of 'Never Split the Difference.' It mentions a scenario involving a CEO

and an employee who is frustrated at his negotiation. Can you tell me the name of the book? The title of the book is 'Never Split the Difference.' Do you know the author of the book? The author of 'Never Split the Difference' is Chris Voss, a former FBI

hostage negotiator. He outlines his strategies for negotiation throughout the book. Impressive! And so that's Project Astra on the phone now. I am one of Google's trusted testers, so there's only a handful of people that have access to this, and it's not going to be

rolled out publicly until after they feel comfortable that all of their tests and red teaming and things like that have gone well, and they are confident to put it out into the public. But this will eventually replace Google's Assistant on your phone. But that's not where Project

Astra ends. The next phase of Project Astra is actually to put these inside of glasses, so I don't need to hold a phone up anymore. I can just look around; it will see what I see and it will respond in my ear in the same way we

were just listening to it. I actually got to test the glasses early. I'm not allowed to talk about it too much yet, but the glasses do exist and they are coming at some point. In fact, here's a little demo of what you might see while using the glasses, because they do have a

Little heads-up display in them. You can see an example here where it's actually giving directions right in front of your eyes as you're walking. Here's another example where they're looking at a menu, and right on the little heads-up display of the screen, it's actually

translating what that menu is saying. Here's somebody else using it to get instructions on how to hang floating shelves, and it is popping up right in front of their eyes inside the glasses. It is wild, honestly. It is really, really cool. And before I forget, I need

to mention some of the things that make Astra really interesting. So again, it's using the Gemini 2.0 model, and the Gemini 2.0 model is multimodal from the ground up. Right? So when you're using, like, ChatGPT and you're talking to it

using the voice mode, it's still using Whisper to transcribe everything you're saying into text, and then it's taking that text before it gives you a response. That's not how Gemini works. Gemini actually understands what it's seeing; it understands what it's hearing; it's not

doing that transcribing underneath. It just understands the audio coming into it. The other thing that makes Astra really interesting is its memory. So it's got a visual memory of up to 10 minutes. So I can ask Astra, "What book were we just talking about a few minutes

ago?" As I recall, we were discussing the book *Never Split the Difference*, and it actually remembers what we were talking about and what book we were looking at. It also remembers things from previous conversations. So I was using this in London last week, so theoretically it should remember what city I was

Exploring last week? In fact, let's see... What city was I exploring last week? You mentioned a trip to London in our earlier conversation, so I assumed that's what you were talking about. However, I apologize for assuming. It's actually got the context and memory of

previous conversations that I had. Personally, I think everybody should have a website. I think it's great to have a place where you can express yourself online. That's why, for today's video, I partnered with Hostinger. Hostinger is a platform where you can host your website,

but you can also completely build your website using AI. If you head on over to hostinger.com/slmtWolf and you claim the special deal, you're going to want to choose the Business Website Builder because this is the one that has all of these cool AI tools built into it. Select

how long you want to set up your hosting for; the longer the period, the less per month it costs. And then see this little link that says 'Have coupon code'? Click that link and type 'Matt Wolf' and click 'Apply,' and you'll see you'll get an

even larger percentage off the plan. And once you're inside of Hostinger, you come over to 'Websites' here, click on 'Add Website,' and choose 'Hostinger Website Builder.' You can see this is an AI-powered drag-and-drop website builder. Let's say, for instance, I want to make a

website all about 3D printers. Let's call the brand name '3D Print World,' and for the description I'll enter something like, 'A website that shares all the latest news and reviews on the newest and most advanced 3D printers.' Once I give it that little bit of detail, I just

Click "Create Website" and let the AI go to work for me.  Within less than a minute, I have a fully fleshed-out website ready to go. It's got images of 3D printers, pre-written articles, and various page sections

like a blog and review. All I have to do is start swapping out the information with what I want on the site. So if I click on "Edit Site" up in the top right, you can see I now have a drag-and-drop builder. I can drag my title around, drag these

buttons around, and this whole website builder has AI baked into it. I can use AI to generate images. Over on the left, I have a whole AI tools menu here. Let's try out the AI heat map. If I click here, you can see it's creating an intention

map, and it uses AI to predict where people's eyeballs are going to go on the website. So as I scroll the website, I get a pretty good idea of what's going to draw people's attention, so that I can strategically place the most important

text and images in the areas that are going to grab attention. This has to be the easiest way to get a website online and running with no technical knowledge at all. So once again, you can check it all out over at hostinger.com/wolf and use the coupon code "MattWolf"

for an additional 10% off their already heavily discounted service. Thank you once again to Hostinger for sponsoring this video. Google also announced Project Mariner this week. Project Mariner is kind of like the computer use that we saw coming out of Claude a few weeks ago.

where you can actually give Google access to do things in your browser. I'm going to start by entering a prompt here. I have a list of outdoor companies listed in Google Sheets, and I want to find their contact information. So I'll ask the agent to take this list of

companies, then find their websites and look up a contact email I can use to reach them. This is a simplified example of a tedious, multi-step task that someone could encounter at work. Now the agent has read the Google Sheet and knows the company names; it then starts

by searching Google for Benchmark Climbing, and now it's going to click into the website. You can see how this research prototype only works in your active tab; it doesn't work in the background. Once it finds the email address, it remembers it and moves on to

the next company. At any point in this process, you can stop the agent or hit pause. What's cool is that you can actually see the agent's reasoning in the user interface, so that you can better understand what it is doing, and it will do the same thing for the next

two companies, navigating your browser, clicking links, scrolling, and recording information as it goes. You're seeing an early-stage research prototype, so we sped this up for demo purposes. After the fourth website, the agent has completed its task, listing out the email addresses for me to use. So this isn't something

I've had the opportunity to play with yet, but as one of the trusted testers, it might be something they put in my hands, and maybe we can experiment with it on a future video. They also showed off Jewels, which is an agent for developers. So Jewels is essentially an assistant that

can help with all of your coding tasks. They also showed off agents inside of games where you can actually be playing a video game on your computer and have it help you with the video game. Another thing that's really interesting about this new Gemini 2.0 model since

it's multimodal from the ground up is that it's got native image output. So you can upload an image, like this car, tell it to turn the car into a convertible, and it will output an image of that car as a convertible. Now that feature doesn't appear to be available yet; I

haven't gotten it to work, at least. But there are also some examples where they blend images. So I thought this looked really cool. You can see they uploaded an image of a cat and a pillow and wrote, 'Create a cross-stitch of my cat on this pillow,' and it created a cross-stitch cat

on that pillow. Here's another example where they put a detailed illustration: 'Sticker of my cat on my skateboard,' and it put a sticker of the cat on the skateboard. So that is really exciting, to be able to take two images and blend them together with a prompt like this.

I'm super excited to get my hands on that feature, but again, I haven't gotten it to work yet, so I don't think it's rolled out just yet. But there's even more news from Google this week; they also rolled out the Deep research

feature inside of Gemini, which is kind of like a more advanced version of Perplexity, where it will actually search the web and do research on topics for you. So if I head back over to my Gemini account, this one I believe is only in the paid plan, so I think you need to

I have Gemini Advanced to work with this one, but the deep research is borderline agentic. It will actually go and do multiple searches and deep dives on a topic for you. So I can give it a prompt like, 'Research quantum computing and how far we are from being able to crack Bitcoin cryptography,' and you'll see it will give us some steps that it's about to take.

So, research websites, find information on the basics of quantum computing, find research papers and articles on the potential of quantum computing to break cryptographic algorithms, etc., etc. Then it's going to analyze the results, create a report, and it will be ready for me in a few minutes.

So if I go ahead and click 'Start research,' it's going to go through all of those steps for me, and we can literally watch it as it goes through and does all this research. So it's researching 16 websites for me right now; now it's up to 45 websites. We can see all of the websites that it's reading from; like, it really goes deep.

Perplexity might use like four or five websites when it's doing the research for you. You can see that Gemini is already up to 65 websites right now. And after several minutes, it came back with a really, really in-depth, detailed report all about quantum computing and its ability to crack the Bitcoin cryptography.

And what's really cool is there's one button here, and I could quickly open it in a Google Doc. So I can quickly save this entire report that it generated, along with all of its source websites, right inside of my Google Doc to refer to it later. Pretty dang cool! And this video is already...

pretty dang cool and this video is already

Getting long, and so far all I've talked about was all of the announcements Google made. And as you probably realized by now, OpenAI has been shipping like crazy as well. They've got their 12 days of OpenAI announcements, and well, since I record these on Thursday, we know what

four more of those announcements are this week. And by the time you're watching this video, there's probably a fifth announcement that came out on Friday. But let's get into the four announcements that OpenAI made this week, starting with the big announcement

that on Monday they released Sora. After demoing it and showing it off 9 months prior, they finally gave us a version of Sora. Now, the version that's out is actually called Sora Turbo, which lets you generate up to 20-second videos if

you're on the Pro plan, and I believe 10-second videos if you're on the Plus plan. And well, since OpenAI waited 9 months between showing it to us and actually getting us access to it, a whole bunch of other alternatives have popped up, and I

don't think Sora made people as excited when they released it as OpenAI had anticipated. Not to mention that OpenAI basically crashed their servers when they opened up Sora because everybody wanted in all at the same time; it couldn't handle the load, and well,

they've since worked that out, and I believe people are able to get in again. But OpenAI's video generator is now finally available. I did a whole video where I tested some prompts because it was so bogged down and it was taking so long for the videos to load; I basically

I had to cut the video shorter than I wanted, and most of the prompts I had tested to that point were all short. What I've learned since then is that Sora really likes long, detailed prompts. I couldn't get it

to generate a wolf howling at the moon in my last demo, but after giving it a much more detailed prompt, I did manage to get a much better wolf howling at the moon video. It really struggles with people doing things like dancing or gymnastics, as you can see in these

examples. But for some reason, it's pretty decent now at a monkey on roller skates. Now, in the Sora video that I released earlier this week, I did go into more detail about some of the features that Sora offers, things like the ability to blend two videos together, like you

can see here, where I took a monkey on roller skates and blended it with clocks flying around. And it's also got a cool storyboard feature where you could try to steer the video a little bit. But in my testing, it wasn't steering

in the direction I wanted it to. I still have to do some more testing. Again, if you want to see me do a deeper dive on Sora, check out this video called "You Can Now Use Sora: Here's How." I dive deeper into some of these

features in that video. On day four of Sora's 12 days of announcements, they released Canvas. Now, if you've had a Plus plan, you've had access to Canvas already. Now they've made Canvas available to everybody; it's on even the free plan.

They also gave the ability to execute Python code inside of Canvas. So if I jump into ChatGPT here, you can see there's a new icon here that says 'View tools,' and it gives us the option for pictures, search, reason, and Canvas. Now I can tell it to write a simple 'Hello

World' code in Python, and just by giving it that prompt, you can see it will generate the code inside of this box. But if I switch to Canvas, give it that same prompt, you'll notice it changes the user interface completely and it puts my chat over on the left and it opens up a sort

of code screen here, and I also have a button that says 'Run,' so I can see the output. And down in the bottom, I can see in my console it says 'Hello, World!' Now, obviously, this is designed for much more complex code. You can also use this

Canvas to do like writing tasks and stuff. So if I create a new chat, make sure I've got Canvas selected, and say, 'Write a poem about wolves on Christmas,' you can see it opens up that same user interface with my prompts on the left and the poem that it's writing here on

the right, along with some additional tools down here in the bottom right, like 'Suggest edits,' 'Adjust the length,' 'Adjust the reading level,' 'Add some final polish,' or, for whatever reason, if you want, you can add emojis as well. So again, Canvas was available already if you were on a

Pro plan; now it's available for free, and now it can also run Python. So that was the day four update. On day five, we learned that it is now available inside of Apple intelligence. If you use Siri, you can actually turn on the ChatGPT

extension inside your Siri, and you can prompt Siri to use ChatGPT when asking it a question. But not only did it roll out on iPhones (and you have to have an iPhone 16 or newer), it also rolled out in macOS as well. If you

have a Mac computer, you can now use ChatGPT with Siri on your Mac computer. You can see they clicked a little Siri icon up in the top right of their Mac, and they can now type to Siri or talk to Siri. Apparently, it can also see your

screen, so they were asking a question about this document, and it took a screenshot and was sharing the screen with ChatGPT using Siri on the actual Apple computer. Now, I feel like this was kind of a sneaky announcement from OpenAI, just because it was really more of an

Apple announcement, but OpenAI made it part of their 12 days of announcements. Because on the same day, the new iOS 18.2 rolled out to all Apple devices with Apple intelligence: the new image playground, Gen emojis, writing tools, and seamless support for ChatGPT and visual

intelligence. So it just kind of seems like they timed their day-five announcement with the same day that they knew Apple was going to be releasing 18.2, so it was kind of a joint company announcement. At this point, most people have probably already seen a lot of the

new Apple intelligence features, like the Gen emojis where you can actually have it create emojis for you, like a DJ octopus or a rainbow cactus. It's got a new and improved note tool where you can use AI directly inside your notes.

Writing tools, the ability to use AI to help you with your text, and we've pretty much seen all of these features from Apple. They're just finally rolling them out in their latest 18.2 update. And then finally, day six, which was Thursday, the day I actually recorded these videos,

announced advanced voice mode now has video capabilities. So this was a feature they showed off seven or eight months ago when they first showed off the advanced voice mode. They actually showed it being able to take pictures and see what you were looking at and have

conversations with you about what you were looking at. When advanced voice mode finally rolled out and was made available, this feature was absent. Now you've got it in your account. To access it, you can just use your ChatGPT account, go into the voice mode by

clicking the little white circle there, and we can see some new options. We have the ability to turn on our camera. We've got the three dots here, which allows us to upload a photo, take a photo, or actually share our screen on our phone. But let's go ahead and test it with some

AI News: Google's AI Can Now SEE Everything!

AI, No-Code, Tech, Futurism - I'm a tech nerd and talk about tech nerd stuff

Matt Wolfe

Google's Gemini 2.0 and AI Studio advancements

Release of Gemini 2.0 Flash, a smaller model outperforming its predecessor.

Free access to Gemini model at gemini.google.com and AI Studio.google.com.

Free use and testing with options for structured output, code execution, and safety settings.

Real-time voice conversation, webcam integration, and screen sharing capabilities.

Demonstration of Gemini's ability to describe the presenter's attire and surroundings.

Spatial understanding features allowing for image analysis and 3D bounding box detection.

Google's Project Astra: AI assistant for phones and future glasses

Project Astra designed as an enhanced AI assistant with visual capabilities.

Demonstration of Astra's ability to identify objects and answer questions based on visual input.

Astra's ability to identify and provide information about a book shown to its camera.

Future integration of Astra into glasses for hands-free operation.

Astra's multimodal capabilities and visual memory features.

Other Google AI announcements: Project Mariner, Jewels, Deep Research

Project Mariner: Browser automation tool.

Gemini 2.0's native image output and image blending capabilities.

Deep Research feature for in-depth web searches.

OpenAI's announcements: Sora, Canvas, Apple integration, and Advanced Voice Mode

Release of Sora Turbo video generation tool.

Canvas update: Availability on free plans and Python code execution.

ChatGPT integration with Apple's Siri on iOS 16 and macOS.

Advanced Voice Mode update: Video capabilities.

Other AI news: Anthropic's Claude 3.5 Haiku, Grok's image generator, Midjourney's Patchwork

Anthropic's Claude 3.5 Haiku model release.

Grok's new image generator using an autoregressive mixture of experts network.

Midjourney's Patchwork collaborative world-building tool.

Adobe's new reflection removal feature in Photoshop and Bridge.

Wider rollout of YouTube's dubbing feature.

Release of Cognition Labs' Devon AI coding assistant.

AI company advertisement in San Francisco.

VR/AR updates: Meta Quest and Windows PC integration, Android XR

Meta Quest 3 integration with Windows PCs for virtual workspace.

Google's Android XR platform and its features.

Tesla robot learning to walk on hills and upcoming live streams

Video of Tesla robot improving its hill-walking capabilities.

Announcement of upcoming live streams to demo AI tools.