Future of Video Games: AI-Generated Real-Time Experiences

Jensen's vision predicts that future video games will be uniquely generated in real-time by AI, transforming content creation entirely.

### The Future of Gaming: AI-Generated Experiences in Real Time
* 🎮  Jensen, the CEO of Nvidia, envisions a future where **every pixel in video games is generated in real time**, creating **unique experiences tailored to individual players**.
* 🕹️  Google Research recently demonstrated this concept by successfully generating the classic game **Doom** using artificial intelligence, showcasing a **significant evolution in game development**.
* 📜  Traditionally, video games are predefined by developers, but the introduction of AI allows for **real-time generation of game environments and mechanics** without the need for pre-written code.
* 🔄  This advancement builds on previous technologies, such as **text-to-image** and **text-to-video** models, ultimately leading to the creation of **fully interactive and generated video games**.
* 📈  The new game engine, developed by Google and its partners, can run Doom at **over 20 frames per second** on a single TPU, demonstrating the potential for **complex game interactions** driven by neural networks.
* 🌐  This shift could revolutionize content creation, allowing users to **request and receive personalized games or shows instantly**, fundamentally altering the role of developers and content creators.
* ⚠️  However, the technology is still in its infancy, facing limitations such as **memory constraints** and the potential for **"hallucinations"** in generated content, which need to be addressed for **future improvements**.
* 🔮  Ultimately, the integration of AI in gaming suggests a future where traditional operating systems may become **obsolete**, as users could simply **ask AI for tailored experiences in real time**.

Jensen, the CEO of Nvidia, said in the future, every single pixel in a video game is going to be generated, not rendered. Imagine you're playing a video game, and it's unlike any other game that anyone else has ever played. It is unique and custom to you. And here's the thing, it's actually being generated in real time. You might think this is some futuristic idea about how video games are going to be, but it's actually a lot closer than you think, and it might already be here. And today, we have a new paper from Google Research that shows that they were able to create the game Doom in a generated way using artificial intelligence. So, I'm going to break this all down for you. I'm going to explain what it means and why it really will change everything about how video games are made. So, first, let's start with a little bit about the game Doom. Doom is a classic video game that came out in the '90s that was absolutely pivotal for its time in terms of graphics and gameplay.

and it's kind of a hacker tradition to put Doom on basically any device. I've seen it run on so many devices, and in fact, it actually has an entire subreddit dedicated to showing off all the different devices that people have gotten Doom to run on. So check this out. Here's one on a phone, here's a price checker, here's a flower pot. I've even seen Doom running on a pregnancy test. And of course, because of that, Doom was the perfect game for Google to show off their new game engine project. So today, the way that video games are made is everything is predefined. A developer or team of developers goes in, they write all the code, all the rules, every single pixel, how it should operate, and they define everything beforehand. And then you render the game and play it, and you're all reading from this game engine. Then we had this evolution in video games in which we had procedural generation, everything from Diablo way back in the day to No Man's Sky more recently. And that basically means that these levels or worlds were not

necessarily predefined, but there was some formula to create them randomly. And now we have finally come to the next step in what video games will be in the future: games that are generated on the fly for you using artificial intelligence. That means that no programmer has written code to define what the game looks like, how it works, any of the rules, none of it. It is being generated in real-time just for you, and that also means you can customize the game as much or as little as you want in real-time as well. And before I break down the paper, I want to show you what has led to this point. We've had text-to-image models for a few years now, so you type in something you want to see, and then the text-to-image model gives you a picture of what you want to see. And then we started having text-to-video models, so you type out what you want to see, and then a video gets output, and nothing in the video was real. It is all generated by artificial intelligence. And then OpenAI released Sora. Sora has been an

incredible achievement meant in the ability to create consistent videos that last minutes simply based on a prompt, and the physics look completely real. A lot of people noticed that some of the videos that Sora was showing off looked really like video games, and so the logical Next Step was actually being able to play these video games, but again, not pre-rendered. They are completely generated. Every single Pixel you see is generated, and now that's what we have. So, this is the paper diffusion models are realtime game engines. This is from Google research, Tel Aviv University, and Google Deep Mind. Take a look at this video, and you will see what I mean. Here we have the game Doom. This is a recreation of the 1993 version, and this is not pre-rendered or rendered whatsoever. This is all generated in real time. All of the monsters coming at you, the movement, walking through the different Halls, all of the interface elements, these are all generated by a neural net. And so, if you thought completely generated content, video games, TV shows, movies was some far-fetched

idea that might happen at some point in the future. Think again, this is happening right now, and this isn't the first example of this, but this is the most sophisticated that I've seen so far. I've seen Call of Duty-like video games that were completely generated using AI, but those games you couldn't actually play. This game you can play. So, let's read the abstract quickly. We present game engine, the first game engine powered entirely by a neural model that enables real-time interaction with a complex environment over long trajectories, so long time periods, at high quality. Game engine can interactively simulate the classic game Doom at over 20 frames per second on a single TPU. And a TPU is basically like a custom GPU. And so, how did they actually do this? Well, the neural net basically predicts what the next frame is going to be at all times. So, if you're moving, it will predict what the next frame is going to be. If you're moving down the hallway, if you're shooting the gun, if

you're killing a monster. It is just constantly predicting exactly like it is predicting the next word in a sentence when you're using a large language model, what the next frame is going to be. So, a game engine is trained in two phases. An RL agent learns to play the game, and the training sessions are recorded. So, they're going to have a bunch of data about what the world looks like and how the player can move throughout the world and what the menu system looks like, what the game's logic and rules are, everything. Then, a diffusion model is trained to produce the next frame conditioned on the sequence of past frames and actions. So, why is this such a big deal? This fundamentally changes the way that all content is going to be created and consumed. When you can generate content in real time, you can generate it for an audience of one. You can describe the exact type of TV show that you want to see, the exact type of video game, how it plays, the style, any cheat codes that you want. You just describe it to AI, and it will create it.

for you, and this opens up the opportunity to have infinite amounts of content that are extremely tailored to an individual. And if you extrapolate this even further, it actually speaks to the future of programming as well. If neural nets and artificial intelligence in general are able to create video games, if they're able to create different types of content, and we've already seen that they're able to create applications, there's really not going to be a need for developers in the future, whether it's video game developers or application developers, and even possibly content creators themselves. And if we really think futuristic, a lot of people have been saying, Sora, that OpenAI text-to-video model that I told you about earlier is truly modeling the real world. So if we're able to model a video game with a neural net, why can't we model the actual world, the real world? And in my mind, we're only limited by the compute that we can throw at it. So this really ends up in a situation like the Matrix if we really just extend out what this

Vision looks like and it speaks to simulation Theory. Now let me just read a couple of the most interesting bits from this research paper. We show that a complex video game, the iconic game Doom, can be run in a neural network, an augmented version of the open stable diffusion 1.4 model, which is great to know in real time while achieving a visual quality comparable to that of the original game. While not an exact simulation, the neural model is able to perform complex game State updates such as tallying health and ammo, attacking enemies, damaging objects, opening doors, and persist the game State over long trajectories. Now here's the thing: it doesn't actually store the state of the game in a database. So if you were to close down that model and then reopen it, it would have no way to know what the state of the game is. It doesn't actually write it to anywhere. It's all done in this neural network. And in the paper, they describe what they believe is a new paradigm for interactive video games. So today video

games are programmed by humans. Game engine is a proof of concept for one part of a new paradigm where games are weights of a neural model, not lines of code. Game engine shows that an architecture and model weights exist such that a neural model can effectively run a complex game interactively on existing hardware. A small part of this vision, namely creating modifications or novel behaviors for existing games, might be achievable in the shorter term. For example, we might be able to convert a set of frames into a new playable level or create a new character just based on example images without having to author code. So, imagine you have a Mario game and you want a brand new world to go explore. You could basically take the existing game and simply prompt a model to give you a new world based on that game, and you could even put yourself in these video games. The possibilities are truly unlimited. Now, let me talk about some of the drawbacks and potential limitations because there definitely are some, and of course, keep in mind this technology is extremely early. We're just

At the beginning, so first it says the game engine suffers from a limited amount of memory. The model only has access to a little over 3 seconds of history, so it's remarkable that much of the game's logic is persisted for drastically longer time horizons. Another limitation is the fact that it is not a perfect simulation of the original game. Now, that only matters if you're trying to recreate an existing game. If you're simply generating a brand new game, that actually doesn't matter, and it's not perfect, just like large language models. This game will hallucinate. Now, the hallucinations come out in different ways, as we can see from this video. There are little twitches in the eyebrow of the avatar. Some of the numbers that are being counted are off. We can see little awkward movements in the graphics, but overall, it does look quite good. So how do we fix all of these issues? Well, first of all, we have to get better at fixing and preventing, in general, hallucinations, which is a broad problem in the world of

artificial intelligence. We need to be able to allow them to persist memory for longer than just a few seconds. And again, this is another problem in the world of AI. AI doesn't currently really have any memory. We use certain memory techniques like retrieval augmented generation to give large language models memory, but they don't actually have memory. They are kind of just frozen in time. And then, of course, we need a lot more training data and a lot more compute. And when we mix all of these things together, I believe the future of video games is truly going to be generated, just like Jensen Wang said early last year. So imagine a future where, rather than waiting for GTA 6, you can simply tell an AI to create it for you. This has huge implications for so many different industries, including movie production, television production, video games, YouTube, of course music, and even computing in general. I truly believe we're not really going to need an operating system or an application layer in the future. We're simply going to ask artificial intelligence for exactly what we need. It's going to

generate an interface and all the data we need in real-time, and then just serve it to us. There really is no purpose to the application layer or even the operating system at that point. The new operating system is artificial intelligence. If you liked this video, please consider giving a like and subscribe, and I'll see you in the next one.

AI DOOM: Fully AI Generated Video Games Are HERE!

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

Nvidia's CEO, Jensen, predicts that future video games will be generated in real-time, offering a unique and custom experience for each player.

Google Research has already demonstrated this concept by using artificial intelligence to create the game Doom in a generated way.

This shift from predefined game development to real-time AI generation could revolutionize how video games are made, allowing for greater customization and flexibility for players.

A new game engine powered entirely by a neural model is revolutionizing real-time video game creation, with the ability to generate every pixel and frame on the fly.

This engine has successfully recreated the classic game Doom, offering interactive gameplay at over 20 frames per second on a single TPU.

The innovative approach of predicting the next frame and using a diffusion model for training is set to change the way content is created and consumed, allowing for personalized experiences on demand.

The Future of Content Creation and Programming

Neural nets and artificial intelligence are poised to create infinite, tailored content, potentially eliminating the need for developers and content creators.

The ability to model complex systems, like video games and possibly the real world, raises questions about the future of programming and simulation theory.

A research paper demonstrates that a neural network can run a complex game like Doom in real time, performing intricate game state updates without storing the game's state in a database.

The Future of Game Development with Neural Models

Game engines are shifting from traditional coding to using neural model weights, showing the potential for interactive game development on existing hardware.

Short-term possibilities include creating modifications or new behaviors for games using example images, potentially offering unlimited creative options.

However, the technology is still in its early stages, with limitations such as memory constraints, imperfect simulations, and occasional hallucinations in the game's logic and graphics.

The Future of AI and Its Impact on Various Industries

AI currently lacks true memory, relying on techniques like retrieval augmented generation.

The future of video game development may involve AI generating content on demand, with implications for various industries such as movie production, music, and computing.

The role of traditional operating systems and application layers may diminish, as AI could provide real-time data and interfaces based on user requests.