New AI Innovations: Text-to-Video and Conversational Models

Genie-2 revolutionizes gaming by generating immersive 3D worlds from a single image, showcasing the future of interactive entertainment.

### Breakthroughs in AI: New Text-to-Video Models and Conversational Agents
* 🎮 Google DeepMind has launched Genie-2, a **groundbreaking text-to-video game model** that generates **fully playable 3D environments** based on a single prompt image, showcasing the **future of gaming**.
* 🤖 Genie-2 enables human or AI agents to interact with its environments using keyboard and mouse inputs, demonstrating its capability through various impressive demos, including **robots navigating different terrains**.
* 🌌 The model features **long horizon memory**, allowing it to remember previously unseen parts of the world and render them accurately when they come back into view, **enhancing the gaming experience**.
* 💡 ElevenLabs has introduced a new conversational AI platform that enables users to create AI agents capable of **natural dialogue**, offering a **seamless way to deploy these agents** across various applications.
* 📚 Additionally, ElevenLabs has launched GenFM, which allows users to **generate personalized podcasts from any text content**, further expanding the **accessibility of AI-driven content creation**.
* 🌐 Tencent has released an open-source text-to-video model named 'Hon', showcasing **high-quality video generation**, while the Quen team has introduced QWQ, an experimental model focused on **enhancing AI reasoning capabilities**.
* 🔗 Prime Intellect has made strides in decentralized AI by releasing Intellect-1, a **10B parameter model** that enables **distributed computing for training**, promoting **open-source collaboration**.
* 💼 Lastly, AWS has entered the AI space with Amazon Qnary Frontier, a multimodal model that **processes vast amounts of data efficiently**, while Anthropic continues to innovate in AI with a significant investment from Amazon to **enhance their capabilities**.

We have a ton of news to go over today. There are so many new text-to-video models and text-to-video game models that have come out, and they are absolutely insane. Let's get right into it. The first story was just released today, as of recording this video; it's

called Genie-2. It is by Google DeepMind and it is a fully playable video game of up to 1 minute, with no underlying game engine. They give a ton of examples, and they are truly mind-blowing. This is the future of video games, and if you

didn't think this is coming soon, think again. Let's take a look. So, Genie-2, a large-scale foundation world model. This is one where, as you could see here, you can actually play them, but they didn't just show screenshots; they gave a ton of demos as well. So I'm going to play some

of these demos as I read more about it. Today we introduced Genie-2, a foundation world model capable of generating an endless variety of action-controllable, playable 3D environments for training and evaluating embodied agents based on a single prompt image. It can be played by a

human or AI agent using keyboard and mouse inputs. Now what I find really cool is that it is a single prompt image that can generate this entire 3D world. And the funny thing is, this is not the only company that came out with something like this this week. I'm going to show

you a new release from World Labs, from Fei-Fei Li and team, in a moment. So let's look at some of these absolutely incredible demos. So here they're showing a bunch of different versions of the same game, with how they react to different controls; so 'W' for forward, 'A'

For left, S for backward, D for right, and space for jump. And as you can see, this is a cool little robot walking through the forest, doing all of those things. Here, in the next one, we have a more sophisticated-looking robot walking through the desert. Here we have a

first-person view of a robot on a purple planet and a first-person view of a robot in a loft apartment in a big city. Again, fully controllable, fully playable for up to 1 minute. And not only that, again from a single frame, they can generate incredibly rich, deep

environments. Look at this example of a boat driving through a lake. So just from that first starting frame, you could see it figures out that, hey, there's sand on the side, and when the boat goes up on the sand it reacts a lot differently. So it knew the physics. Now, obviously, it

doesn't look perfect, but imagine what it's going to look like just six months from now, a year from now, two years from now. This is going to be the worst it ever is. Here's another example of kind of like a more fantasy game, walking

through the forest, and it has what it's calling long horizon memory. Genie 2 is capable of remembering parts of the world that are no longer in view and then rendering them accurately when they become observable again. This reminds us back to when we first saw Sora; that was

really the first time where we saw objects disappear behind different objects in the foreground, only to reappear perfectly in place where you would have expected them. Later in the video, and now we have a dynamic video game version of that. So look at this: on the left here, you could see a

A little character running around the entire world is being predicted and remembered. Same here; we have a robot running through the pyramids, and you can see as it goes out of frame, it comes back into frame exactly where it was. Same with this kind of futuristic, cyberpunk...

punkish robot walking around. You could see all the buildings staying in line where they belong. Thank you to the sponsor of this video: Build Your Store AI. Starting an online business has never been easier, with no upfront costs, minimal investment, and a manageable time commitment. It's an ideal way to...

make some money on the side, and of course, you can scale it over time if it works. So, whether you're looking to supplement your income or turn it into a full-time job, an online business offers that flexibility and significant growth potential. So if you're ready to...

get started with e-commerce, selling stuff online, Build Your Store AI, or BYAI, is a perfect solution. Build Your Store AI allows you to create an entire storefront by just describing what you want using AI. Let me show you how easy it is to set up Build Your Store with...

Shopify. Register and choose a niche. Set up your Shopify account, and you can get started for just $1 per month for the first three months. Install the Build Your Store app into Shopify, then review your store, make sure it looks good, make any changes that you want, and publish it.

You're good to go! You can follow these steps and get started really quickly. Test out a bunch of ideas, and when you find one that works, scale it up. That is the name of the game, and really, you didn't have to do any coding yourself or...

Any web design yourself, so if you want to check it out, I'll drop links in the description below. And thanks again to the sponsor of this video, Build Your Store AI, and now back to the video. Here are some more examples. I think this is one of the best-looking examples; this is

kind of like an RPG-style game where we have this little character running around a town. Here is a gorgeous racing game that I think works really well. Here's one of a samurai walking through kind of a green field with a bunch of balloons, and it's actually able to chop

through a balloon, and the balloon explodes like you think it would. Here's a first-person shooter where it's shooting a barrel of gasoline, and the barrel explodes, and you can actually see the physics of the barrel post-explosion. Here's one where he's climbing a ladder

and just like you would expect: one hand, one hand, one foot, one foot. Looks very realistic. And again, I have to remind you, no underlying game engine. These are all predictive engines, meaning there's no underlying game engine; it's just using AI diffusion models, most likely, and it

is predicting what the next frame in the series will be. Here's two examples of smoke. So, on the left side, it kind of looks like old-school London smoke coming out of chimneys; right there, they look really good. Here's one; looks like a first-person shooter, a volcano, plenty of

smoke coming out of the volcano, all looking very realistic. Here's some lighting, which I think is probably one of the most impressive demos that they have. Here's kind of maybe a Zelda-esque game, walking through this little dungeon, and the fire he's holding lights up the

The entire scene is very realistic, and you can drop real-world images and start playing from those. So these are just real-world images placed in the model, and then it creates the world from that. The cool part is you can go from concept art to a playable game just to see how it feels.

Instantly! Here's an environment concept by Max Can't, and I dropped it into Genie, and you can immediately play it and see how it feels. Same thing right here; another environment concept, and boom! You can actually see all the different elements that were in the concept in the

predicted model. So there are a ton of other examples on this website. I highly recommend you check it out. I'll drop the link in the description below. Definitely one of the coolest things that I've seen in the world of AI; one of the things I'm most excited about in the

coming years is to see how video games change forever. And as I said, they're not the only company that released an incredible demo and update about world modeling—and that's essentially what we're talking about—and the company World Labs, which I did a couple of videos

on. This is one started by a small team. We've been busy building an AI system to generate 3D worlds from a single image, so very similar to what we've been talking about. Check out some of the early results on our site where you can

interact with our scenes directly in the browser. So I'm going to show you a few examples, then I'm going to show you playing around with it. It's definitely different and maybe not as in-depth as the previous example from Google that I just showed you, but it's still

Impressive nonetheless, and they actually give you playable demos, which Google does not. And here's the cool thing: you can actually control the scene directly using camera angles, like you would in a traditional game engine. Here's what they say: most generative models predict pixels; predicting a 3D scene instead has

many benefits. The scene won't change if you look away and come back, and it obeys the basic physical rules of 3D geometry. The simplest way to visualize the 3D scene is a depth map, where each pixel is colored by its distance to the camera.

And so that's what we're seeing right there. And so they're taking a very different approach where, instead of predicting each pixel, they are predicting the entire scene together as one. Here's another example: because of the way that they're generating the entire scene, you can actually change

things in the scene, which is very novel. So you can change the lighting easily; you can add spotlights, as you're seeing here, all of this done in real time. You can also expand from single frames; so this is a very famous painting, and you can move back; you could start to look

around at the rest of the street, which is insane to think about, and it's all perfectly rendered. Looks awesome, and all of the physics look very real. Okay, so here's one of the examples; let me just show you what it looks like. It's pretty

limited in what the demo allows you to do right now, but still very cool nonetheless. So here we can look around this little world; we can look around in pretty much any direction, but we can only move a few feet before it says 'out of bounds'. So still very cool. Let me show

You know, one more example; this one's gorgeous. Look at this! So you can look in any direction, 360 degrees. We can move only a few steps before it says 'out of bounds.' So you could check out all of these examples; you can play around with it. I'll drop the links in

the description below. Now, moving on from 3D worlds, we are going to talk about conversational AI agents. Now, the most impressive one we know so far is Advanced Voice from OpenAI, but now ElevenLabs has something that is seemingly on par with Advanced Voice mode. So ElevenLabs

launched conversational AI. Build AI agents that can speak in minutes with low latency, full configurability, and seamless scalability. Let's take a listen to what this sounds like:  Building AI agents that can speak is now easier than ever.  'Hi, what would you like to talk

about today? Introducing conversational AI with ElevenLabs. Build, test, and deploy all in one platform for the most natural way to communicate with technology.' 'Hey, can I check the status of my order?' 'Yes, of course. Let me check for you now. Your order is due to arrive at around 2 p.m.'

Building conversational AI agents used to be immensely complex and demand huge resources, but not anymore. Start building your agent by creating a voice or exploring our vast, high-quality library to discover one that's perfect for your application. 'Hello and welcome to DHK online. How can I help you today?' With a

wide range of the best LLMs to choose from, you can then upload your knowledge base and define the goal and personality of your agent, or alternatively, you can integrate your own server to take full control. 'Hi, this is Eric from Bound Events. How can I help?' 'Hey, I accidentally

Uh, I bought a ticket twice. Could I get a refund, please?

Sure, please read out your reference number and I can begin to process that right now. Our system will analyze and evaluate your transcripts, providing you with valuable insights, conversation playback, and the ability to test and define success criteria.

It's all possible in 32 different languages. Welcome to DC Medical. How can I help?

Hello. And best yet, effortlessly deploy your agent into websites with a single copy and paste, or leverage our SDK for creating apps, services, or even video games. So whatever you're creating, get to

production in days, not months, and all with enterprise-grade security to keep your users' data safe and secure. It's time to explore the huge potential of conversational AI with ElevenLabs. So I think the one difference here between this and advanced voice mode is:

Advanced voice mode is an end-to-end voice model, so it is not actually translating back and forth to text. However, for ElevenLabs, I believe it is. So the process seems to be: it takes the voice, transcribes it to text, uses an LLM to figure out what to

Want, and you could try different LLMs based on what is best for your use case. And if conversational voice agents weren't enough, ElevenLabs also essentially dropped a copy of Google's NotebookLM product—remember that one? Very cool product; kind of went viral. It's

essentially, you can load up any piece of content, text-based content, and make it into a podcast, and it sounded really cool. And now ElevenLabs lets you do that as well. So you can now generate smart, personalized podcasts from any of your PDFs, articles, ebooks, links, or text in 32

languages with our ElevenReader iOS app. So here's an example of what that sounds like: "And as for Cinderella and the prince, they lived happily ever after." There is always more to every story, and this is a brand new way to experience [Music]

it. Introducing our new GenFM podcast feature. Say hello to your hosts. I mean, seriously, I find Cinderella just so inspiring as a character throughout this whole book. Yeah, I know, right? And she does it all in those tiny glass slippers—unreal! Now you can generate your own

podcast all at the click of a button. All right, so let's dive into today's review topic: atomic theory. Now, this is one of my absolute favorite subjects. Where should we begin? Choose any topic, any voices, any time. Next, the recipe says to lay out the apple slices until the pan

is full. Ah, yes, this is a true classic of fine French cuisine. GenFM, now available in your pocket with the ElevenReader. So very, very cool. Love all of this competition in the AI space. Congratulations to two incredible launches by ElevenLabs. Next, we

Have another fully open-source text-to-video model. This is by Tencent; it's called — I hope I'm not getting this wrong —  'Hon.' And it looks really cool, and it's open source; open weights. You can download it here. Here are some really cool examples. This looks like Halo underwater.

environment. Let's look a little bit about it. So here's one; very high quality. Now, all of the clips are pretty short; they're all about 5 seconds, and I bet most of them are going to be about that length. Now here's another one: panda

riding a bicycle in a busy street. Now I just did a tutorial, which should have dropped by the time you're seeing this video. It's another text-to-video model called 'Mochi' that you can download and get running on your local computer. And this is just another example; another

really cool example of open-source text-to-video. Now that one was only a few seconds as well. Obviously, you could do it much longer, but you need a lot more computing power and a lot more time. But so, what? This is awesome! Here's another one. Definitely some uncanny valley in

this one; the eyes are awkward, the lips are weird, but fine, looks good still. And here we go; a bit of physics. This is liquid physics; looks good to me. One more example that I think looks really, really good, with some sound effects. This is a — what looks like — a Ford F-150 Raptor

going through the mud. And most of the physics looks really good; the mud looks great. I mean, it's just very impressive. So you can read all about it; they published a lot of information about it. I'm not going to go through it in this video. If you want to see me do a

Tutorial about downloading this, getting it running on your computer. Let me know in the comments; you know I like doing that. And speaking of open-source models coming out of China, we have another incredible model. This is a quote-unquote "thinking model" called QWQ, and this is by the Quen team, and so...

they released this. It is open source, open weights, and that means you could download it. And this is a true thinking model. Now there's a lot of caveats here. The creator said it still behaves quite weirdly in some cases, but incredibly, mind-blowingly, in other cases. So it's

far from perfect, but they got it out. They wanted it to be tested, and they're obviously going to continue to iterate on it. I appreciate anybody who contributes to the open-source AI community. So QWQ: Reflect deeply on the boundaries of the unknown. QWQ 32B

preview. So it's a 32-billion parameter model, so it should fit on at least high-end machines.  It's an experimental research model developed by the Quen team, focused on advancing AI reasoning capabilities. As a preview release, it demonstrates promising analytical abilities while having several important

limitations: Language mixing and code switching—the model may mix languages or switch between them unexpectedly; recursive reasoning loops—the model may enter circular reasoning patterns, leading to lengthy responses without a conclusive answer; safety and ethical considerations—the model requires enhanced safety measures to ensure

reliable and secure performance; performance and benchmark limitations—the model excels in math and coding but has room for improvement in other areas, such as common sense reasoning and nuanced language understanding. So definitely performs well; it actually beats the 001 preview by OpenAI in the

Aim, which is math, I believe. Benchmark GPT QA; it comes in under, beats it in the math 500, and the live code bench is about even. So this is great performance, especially for a preview model. So let's look at an example of what the actual reasoning looks like so

Please add a pair of parentheses to the incorrect equation, and here it is right there, to make the equation true. All right, so here we go. It starts thinking. The cool thing is you can actually see all of the chain of thought, which is very unlike the 01 model, which hides all

of that. Let's tackle the problem step by step. Given this equation, it's incorrect; I need to add parentheses. So, calculating step by step using PEMDAS and so on, it starts to add parentheses and... very long thinking, lots of thinking, lots of going back and forth, and then finally let's

see what the output is and final answer. So there it is. So a lot of thinking. I'm not sure how long it took; it doesn't actually say, as far as I can tell, but yeah, it did just fine and it thought through all of it and gave a good answer.

So definitely check it out. Of course, you know, I'm going to ask: Do you want me to test it? Let me know in the comments. Okay, next, moving on from thinking models, we are now moving on to decentralized trained models. That means it may no longer be required to have massive data

centers used to train these next-generation models. We could do something like how torrents work and distribute the compute amongst many, many smaller machines across the entire world. Now I did a video about this with a project called Pedals a while back, probably about a year ago, and this is

Really, the first time I've seen any other project that's even similar, and it seems like they made it work. So this is by Prime Intellect; they have released Intellect-1, an open-source, decentralized, 10B parameter model, and they give a technical paper with all of the details.

And they also give the repo, and you can actually try the model if you'd like. I'm not going to get into that right now, but very cool, because this is going to vastly help the open-source community. Typically, there aren't a lot of companies

that are spending billions and billions of dollars, aside from Meta, that are training open-source models. They train the models, they put them out there, and honestly, Meta could change their mind at any moment; that's their right. But if we have tools like this, we, the open-source

community, then we're going to be able to potentially train our own models. So I have two massive RTX A6000s in this machine next to me; they mostly sit dormant throughout the day. I could just give a portion of that compute towards

this type of project and help train the next generation of open-source models. Very cool, in my mind; big innovation. A lot of people did not think that we'd be able to split up the compute of these models and then kind of put it all together at the end, but

here's an example of a 10B parameter model that did it. So congrats to Prime Intellect; great, great contribution. Next, from Anthropic, this was a big announcement: they have introduced the Model Context Protocol, MCP, and this is a new way to give agents access to real-world tools. This is a

A protocol is a set of standards that standardizes how agents interact with the web, browsers, and computers. It's a new standard for connecting AI assistants to systems where data resides, including content repositories, business tools, and development environments.

Its aim is to help frontier models produce better, more relevant responses.  It seems most frontier AI companies are working on this, whether it's rumors of OpenAI building its own browser or their operator feature (hopefully coming next year), or Claude's computer control that just came out.  Anthropic now has the MCP, so it seems many companies are working on this—the next generation of agents having standardized ways to interact with the real digital world.

Model Context Protocol is an open standard enabling developers to build secure, two-way connections between their data sources and AI-powered tools. The architecture is straightforward; developers can expose their data through MCP servers or build AI applications that connect to these servers.

Claude 3.5 is adept at quickly building MCP server implementations, making it easy for organizations and individuals to rapidly connect their most important datasets with a range of AI-powered tools. They've already released pre-built ones for Google Drive, Slack, GitHub, Git, PostgreSQL, and Puppeteer.

This is something I've been saying for a long time: the future of computer interaction is humans talking to an agent that gets you exactly what you want.

I believe that the application layer, as it is today, is going to disappear. There really is no need for it. I think any company that can really just be kind of boiled down to a UI layer on top of a database—Salesforce is probably most ripe to be disrupted by something like

this—because what is the point of having the UI when the agent can generate the UI on the fly, based on the data that it can get directly from the database? That's my belief. I don't think it's a year or two down the road; I think we're

at least 5 or 10 years from that, but still, that is a fundamental change for how SaaS has operated for a very long time. So check it out; as always, I'll drop all the links in the description below. All right, now for kind of a cool side

project from Google. This is called Gen Chess, and you can basically generate chess sets based on anything. These are generative chess sets. I haven't really shared this, but I absolutely love chess. I love playing it; I love watching chess streamers; I love watching the

professionals play. I was really excited to see this. So here's a creative chess set inspired by ice cream, and you can change it to be anything you want. So I'll say, 'inspired by robots,' and generate. So cool little toy project! I think it's awesome. Very cool robots, and you could

tap to edit, and you can edit any which way you like. You can see different variations of it—all the different pieces. Very cool project, Google! Really good job. Honestly, Google has had some very impressive releases these last two weeks. All right, next we have a new IM

Runway, the company best known for text-to-video, now has a text-to-image model. Here are some examples; they look phenomenal—absolutely phenomenal. Frames is their newest base model for image generation, marking a big step forward in stylistic

control and visual fidelity. The model excels at maintaining stylistic consistency while allowing for broad creative exploration. Frames can establish a specific look for your project and reliably generate variations that stay true to your aesthetic. So here are just some examples that come out of

this new model. This is probably one of the most impressive text-to-image models I've seen, just in terms of quality and realism, and honestly, this stylistic vibe. It's not just realism; it's more like something you're

going to see in a movie, and I think that is very much aligned with how Runway sees itself—aligning themselves with the movie industry. So, a bunch of different stuff here: 1970s album art, very cool; creepy stuff; Japanese Zen; here we go.

Look at these beautiful digital picture profiles! I mean, look at this guy; look at the details. I can't see a single thing wrong with this. If I saw this in a magazine, I would never know the difference. Here are some nature shots, and again, if I saw this in a magazine,

I wouldn't know the difference. Here are some really cool disposables—so this is like something you're going to take on a disposable camera; it really has that very grainy look—but I love this aesthetic. Yeah, just unbelievable, unbelievable stuff. So very cool, Runway.

All right, now from text to image to text to text, we have a new LLM from AWS. They are finally putting their foot out there, getting their own model out to the world. So they've introduced Amazon Qnary Frontier, an intelligence and industry-leading price-performance model, as far as I know, and it seems to be incredibly cheap, though, and performs very, very well. It not only has text, but it can understand vision as well.

They have three different sizes: micro, light, and pro. Micro has a context length of 128k tokens. The light version is a very low-cost multimodal model that is lightning fast for processing image, video, and text inputs to generate text outputs. 30 minutes of video in a single request! We are getting up to what we've seen from Google with their 2 million context window and being able to really

ingest hours of video. Pro processes 300k input tokens. Very, very cool! They do have a fourth one: Amazon Qnary Premiere, our most capable multimodal model for complex reasoning tasks and for use as the best teacher for distilling custom models. Amazon Qnary Premiere is still in

training, targeting early 2025, so that one's not out yet. So all of this is available in Amazon Bedrock. It is very good to see AWS getting into their own models. And speaking of Amazon and their own models and AI in general, they are continuing to expand their investment

into Anthropic. So Anthropic announced, "We're expanding our collaboration with AWS. This includes a new $4 billion investment from Amazon and establishes AWS as our primary cloud and training partner." They are developing future generations of Claude chips, designing both hardware and software together. Let's

Genie-2, World Labs Demo, Hunyuan Video, Distributed Training, Amazon Nova, 12 Days of OpenAI

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

New Text-to-Video and Text-to-Video Game Models

Introduction to numerous new text-to-video models and their impact.

Genie-2 by Google DeepMind: fully playable video games up to 1 minute long, without a game engine, using a single prompt image.

Genie-2 demos showcasing various game environments and control responsiveness.

Genie-2's ability to generate realistic physics and long horizon memory.

Diverse game examples generated by Genie-2, including RPG, racing, and first-person shooter styles.

World Labs' AI system generates 3D worlds from a single image, offering browser-based interaction.

World Labs' approach focuses on predicting the entire 3D scene, not just pixels, resulting in scene consistency and adherence to physical rules.

World Labs' ability to modify scenes in real-time, changing lighting and adding elements.

Interactive demo showcasing World Labs' technology, with limitations in movement area.

ElevenLabs' Conversational AI and Podcast Generation

ElevenLabs launches conversational AI, allowing creation of AI agents that can speak with low latency and scalability.

Examples of ElevenLabs' conversational AI in action, showcasing natural communication with technology.

ElevenLabs' text-to-podcast feature, similar to Google's NotebookLM, enabling podcast creation from various text-based content.

Audio example of ElevenLabs' podcast generation feature.

Tencent's open-source text-to-video model, 'Hon', with examples showcasing high-quality video generation.

Discussion of another open-source text-to-video model, 'Mochi', and its accessibility for local computer use.

Quen team's open-source 'thinking model,' QWQ, focusing on AI reasoning capabilities with limitations.

Example of QWQ's reasoning process, showcasing its step-by-step problem-solving approach.

Decentralized Model Training and Anthropic's MCP

Introduction to decentralized model training, distributing computation across smaller machines.

Prime Intellect's Intellect-1: an open-source, decentralized 10B parameter model.

Anthropic's Model Context Protocol (MCP): a new standard for connecting AI assistants to real-world tools and data sources.

MCP's pre-built integrations with various services like Google Drive, Slack, and GitHub.

Google's Gen Chess: a tool for generating custom chess sets based on various themes.

Runway's new text-to-image model, Frames, showcasing high-quality and realistic image generation.

AWS introduces Amazon Qnary Frontier, a multimodal LLM with different sizes and capabilities.

Amazon's continued investment in Anthropic, focusing on the development of future generations of Claude chips.

Sam Altman's announcement of "12 Days of Open AI," promising daily updates and releases.