비디오 게임의 미래: AI로 생성된 실시간 경험

### 게임의 미래: 실시간으로 생성되는 AI 경험
* 🎮 엔비디아의 CEO인 젠슨은 **비디오 게임의 모든 픽셀이 실시간으로 생성**되어 개별 플레이어에게 맞춤화된 독특한 경험을 선사하는 미래를 상상합니다.
* 🕹 구글 리서치는 최근 인공지능을 사용하여 **고전 게임 둠을 성공적으로 생성**함으로써 이 개념을 입증하여 게임 개발의 중요한 진화를 보여주었습니다.
* 📜 전통적으로 비디오 게임은 개발자가 미리 정의했지만, **AI를 도입하면 미리 작성된 코드 없이도 게임 환경과 메커니즘을 실시간으로 생성**할 수 있습니다.
* 🔄 이러한 발전은 **텍스트 대 이미지 및 텍스트 대 비디오 모델**과 같은 이전 기술을 기반으로 하여 궁극적으로 **완전한 인터랙티브 및 생성 비디오 게임 제작**으로 이어집니다.
* 📈 Google과 파트너가 개발한 새로운 게임 엔진은 **단일 TPU에서 초당 20프레임 이상의 속도로 둠**을 실행할 수 있으며, 신경망으로 구동되는 복잡한 게임 상호작용의 잠재력을 보여줍니다.
* 🌐 이러한 변화는 콘텐츠 제작에 혁명을 일으켜 **사용자가 개인화된 게임이나 프로그램을 즉시 요청하고 받을 수 있게** 함으로써 개발자와 콘텐츠 제작자의 역할을 근본적으로 변화시킬 수 있습니다.
* ⚠ 그러나 이 기술은 아직 초기 단계로, **메모리 제약과 생성된 콘텐츠의 '환각' 가능성** 등의 한계에 직면해 있으며 향후 개선을 위해 해결해야 할 과제가 남아 있습니다.
* 🔮 궁극적으로 게임에 AI가 통합되면 사용자가 실시간으로 AI에 맞춤형 경험을 요청할 수 있기 때문에 **기존 운영 체제가 쓸모없어질 수 있는 미래**가 도래할 수 있습니다.

젠슨의 비전에 따르면 미래의 비디오 게임은 AI에 의해 실시간으로 고유하게 생성되어 콘텐츠 제작을 완전히 변화시킬 것입니다.

Jensen, the CEO of Nvidia, said in the future, every single pixel in a video game is going to be generated, not rendered. Imagine you're playing a video game, and it's unlike any other game that anyone else has ever played. It is unique and custom to you. And here's the thing, it's actually being generated in real time. You might think this is some futuristic idea about how video games are going to be, but it's actually a lot closer than you think, and it might already be here. And today, we have a new paper from Google Research that shows that they were able to create the game Doom in a generated way using artificial intelligence. So, I'm going to break this all down for you. I'm going to explain what it means and why it really will change everything about how video games are made. So, first, let's start with a little bit about the game Doom. Doom is a classic video game that came out in the '90s that was absolutely pivotal for its time in terms of graphics and gameplay.

and it's kind of a hacker tradition to put Doom on basically any device. I've seen it run on so many devices, and in fact, it actually has an entire subreddit dedicated to showing off all the different devices that people have gotten Doom to run on. So check this out. Here's one on a phone, here's a price checker, here's a flower pot. I've even seen Doom running on a pregnancy test. And of course, because of that, Doom was the perfect game for Google to show off their new game engine project. So today, the way that video games are made is everything is predefined. A developer or team of developers goes in, they write all the code, all the rules, every single pixel, how it should operate, and they define everything beforehand. And then you render the game and play it, and you're all reading from this game engine. Then we had this evolution in video games in which we had procedural generation, everything from Diablo way back in the day to No Man's Sky more recently. And that basically means that these levels or worlds were not

necessarily predefined, but there was some formula to create them randomly. And now we have finally come to the next step in what video games will be in the future: games that are generated on the fly for you using artificial intelligence. That means that no programmer has written code to define what the game looks like, how it works, any of the rules, none of it. It is being generated in real-time just for you, and that also means you can customize the game as much or as little as you want in real-time as well. And before I break down the paper, I want to show you what has led to this point. We've had text-to-image models for a few years now, so you type in something you want to see, and then the text-to-image model gives you a picture of what you want to see. And then we started having text-to-video models, so you type out what you want to see, and then a video gets output, and nothing in the video was real. It is all generated by artificial intelligence. And then OpenAI released Sora. Sora has been an

incredible achievement meant in the ability to create consistent videos that last minutes simply based on a prompt, and the physics look completely real. A lot of people noticed that some of the videos that Sora was showing off looked really like video games, and so the logical Next Step was actually being able to play these video games, but again, not pre-rendered. They are completely generated. Every single Pixel you see is generated, and now that's what we have. So, this is the paper diffusion models are realtime game engines. This is from Google research, Tel Aviv University, and Google Deep Mind. Take a look at this video, and you will see what I mean. Here we have the game Doom. This is a recreation of the 1993 version, and this is not pre-rendered or rendered whatsoever. This is all generated in real time. All of the monsters coming at you, the movement, walking through the different Halls, all of the interface elements, these are all generated by a neural net. And so, if you thought completely generated content, video games, TV shows, movies was some far-fetched

idea that might happen at some point in the future. Think again, this is happening right now, and this isn't the first example of this, but this is the most sophisticated that I've seen so far. I've seen Call of Duty-like video games that were completely generated using AI, but those games you couldn't actually play. This game you can play. So, let's read the abstract quickly. We present game engine, the first game engine powered entirely by a neural model that enables real-time interaction with a complex environment over long trajectories, so long time periods, at high quality. Game engine can interactively simulate the classic game Doom at over 20 frames per second on a single TPU. And a TPU is basically like a custom GPU. And so, how did they actually do this? Well, the neural net basically predicts what the next frame is going to be at all times. So, if you're moving, it will predict what the next frame is going to be. If you're moving down the hallway, if you're shooting the gun, if

you're killing a monster. It is just constantly predicting exactly like it is predicting the next word in a sentence when you're using a large language model, what the next frame is going to be. So, a game engine is trained in two phases. An RL agent learns to play the game, and the training sessions are recorded. So, they're going to have a bunch of data about what the world looks like and how the player can move throughout the world and what the menu system looks like, what the game's logic and rules are, everything. Then, a diffusion model is trained to produce the next frame conditioned on the sequence of past frames and actions. So, why is this such a big deal? This fundamentally changes the way that all content is going to be created and consumed. When you can generate content in real time, you can generate it for an audience of one. You can describe the exact type of TV show that you want to see, the exact type of video game, how it plays, the style, any cheat codes that you want. You just describe it to AI, and it will create it.

for you, and this opens up the opportunity to have infinite amounts of content that are extremely tailored to an individual. And if you extrapolate this even further, it actually speaks to the future of programming as well. If neural nets and artificial intelligence in general are able to create video games, if they're able to create different types of content, and we've already seen that they're able to create applications, there's really not going to be a need for developers in the future, whether it's video game developers or application developers, and even possibly content creators themselves. And if we really think futuristic, a lot of people have been saying, Sora, that OpenAI text-to-video model that I told you about earlier is truly modeling the real world. So if we're able to model a video game with a neural net, why can't we model the actual world, the real world? And in my mind, we're only limited by the compute that we can throw at it. So this really ends up in a situation like the Matrix if we really just extend out what this

Vision looks like and it speaks to simulation Theory. Now let me just read a couple of the most interesting bits from this research paper. We show that a complex video game, the iconic game Doom, can be run in a neural network, an augmented version of the open stable diffusion 1.4 model, which is great to know in real time while achieving a visual quality comparable to that of the original game. While not an exact simulation, the neural model is able to perform complex game State updates such as tallying health and ammo, attacking enemies, damaging objects, opening doors, and persist the game State over long trajectories. Now here's the thing: it doesn't actually store the state of the game in a database. So if you were to close down that model and then reopen it, it would have no way to know what the state of the game is. It doesn't actually write it to anywhere. It's all done in this neural network. And in the paper, they describe what they believe is a new paradigm for interactive video games. So today video

games are programmed by humans. Game engine is a proof of concept for one part of a new paradigm where games are weights of a neural model, not lines of code. Game engine shows that an architecture and model weights exist such that a neural model can effectively run a complex game interactively on existing hardware. A small part of this vision, namely creating modifications or novel behaviors for existing games, might be achievable in the shorter term. For example, we might be able to convert a set of frames into a new playable level or create a new character just based on example images without having to author code. So, imagine you have a Mario game and you want a brand new world to go explore. You could basically take the existing game and simply prompt a model to give you a new world based on that game, and you could even put yourself in these video games. The possibilities are truly unlimited. Now, let me talk about some of the drawbacks and potential limitations because there definitely are some, and of course, keep in mind this technology is extremely early. We're just

At the beginning, so first it says the game engine suffers from a limited amount of memory. The model only has access to a little over 3 seconds of history, so it's remarkable that much of the game's logic is persisted for drastically longer time horizons. Another limitation is the fact that it is not a perfect simulation of the original game. Now, that only matters if you're trying to recreate an existing game. If you're simply generating a brand new game, that actually doesn't matter, and it's not perfect, just like large language models. This game will hallucinate. Now, the hallucinations come out in different ways, as we can see from this video. There are little twitches in the eyebrow of the avatar. Some of the numbers that are being counted are off. We can see little awkward movements in the graphics, but overall, it does look quite good. So how do we fix all of these issues? Well, first of all, we have to get better at fixing and preventing, in general, hallucinations, which is a broad problem in the world of

artificial intelligence. We need to be able to allow them to persist memory for longer than just a few seconds. And again, this is another problem in the world of AI. AI doesn't currently really have any memory. We use certain memory techniques like retrieval augmented generation to give large language models memory, but they don't actually have memory. They are kind of just frozen in time. And then, of course, we need a lot more training data and a lot more compute. And when we mix all of these things together, I believe the future of video games is truly going to be generated, just like Jensen Wang said early last year. So imagine a future where, rather than waiting for GTA 6, you can simply tell an AI to create it for you. This has huge implications for so many different industries, including movie production, television production, video games, YouTube, of course music, and even computing in general. I truly believe we're not really going to need an operating system or an application layer in the future. We're simply going to ask artificial intelligence for exactly what we need. It's going to

generate an interface and all the data we need in real-time, and then just serve it to us. There really is no purpose to the application layer or even the operating system at that point. The new operating system is artificial intelligence. If you liked this video, please consider giving a like and subscribe, and I'll see you in the next one.

AI DOOM: Fully AI Generated Video Games Are HERE!

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

엔비디아의 CEO인 젠슨은 미래의 비디오 게임은 실시간으로 생성되어 각 플레이어에게 고유하고 맞춤화된 경험을 제공할 것이라고 예측합니다.

구글 리서치는 이미 인공 지능을 사용하여 게임 둠을 생성 방식으로 제작함으로써 이 개념을 입증했습니다.

사전 정의된 게임 개발에서 실시간 AI 생성으로의 전환은 비디오 게임 제작 방식을 혁신하여 플레이어에게 더 큰 커스터마이징과 유연성을 제공할 수 있습니다.

뉴럴 모델로 구동되는 새로운 게임 엔진은 모든 픽셀과 프레임을 즉시 생성하는 기능을 통해 실시간 비디오 게임 제작에 혁신을 불러일으키고 있습니다.

이 엔진은 고전 게임인 둠을 성공적으로 재현하여 단일 TPU에서 초당 20프레임 이상의 인터랙티브 게임플레이를 제공합니다.

다음 프레임을 예측하고 훈련에 확산 모델을 사용하는 혁신적인 접근 방식은 콘텐츠 제작 및 소비 방식을 변화시켜 필요에 따라 개인화된 경험을 제공할 수 있도록 합니다.

콘텐츠 제작 및 프로그래밍의 미래

신경망과 인공 지능은 무한한 맞춤형 콘텐츠를 만들어낼 준비가 되어 있어 잠재적으로 개발자와 콘텐츠 제작자의 필요성을 없애줄 수 있습니다.

비디오 게임이나 현실 세계와 같은 복잡한 시스템을 모델링하는 능력은 프로그래밍과 시뮬레이션 이론의 미래에 대한 의문을 제기합니다.

한 연구 논문은 신경망으로 둠과 같은 복잡한 게임을 실시간으로 실행하여 게임 상태를 데이터베이스에 저장하지 않고도 복잡한 게임 상태 업데이트를 수행할 수 있음을 보여줍니다.

뉴럴 모델을 활용한 게임 개발의 미래

게임 엔진은 기존의 코딩 방식에서 신경 모델 가중치를 사용하는 방식으로 전환하고 있으며, 기존 하드웨어에서 인터랙티브 게임을 개발할 수 있는 가능성을 보여주고 있습니다.

단기적인 가능성으로는 예제 이미지를 사용하여 게임의 수정 또는 새로운 동작을 생성하는 것이 있으며, 잠재적으로 무한한 창의적 옵션을 제공할 수 있습니다.

그러나 이 기술은 아직 초기 단계에 있으며 메모리 제약, 불완전한 시뮬레이션, 게임 로직과 그래픽에서 가끔 발생하는 환각과 같은 한계가 있습니다.

AI의 미래와 다양한 산업에 미치는 영향

현재 AI는 진정한 메모리가 부족하여 검색 증강 생성과 같은 기술에 의존하고 있습니다.

비디오 게임 개발의 미래에는 영화 제작, 음악, 컴퓨팅 등 다양한 산업에 영향을 미치는 온디맨드 콘텐츠를 생성하는 AI가 등장할 수 있습니다.

AI가 사용자 요청에 따라 실시간 데이터와 인터페이스를 제공할 수 있기 때문에 기존 운영 체제와 애플리케이션 계층의 역할이 줄어들 수 있습니다.