what is spatial artificial intelligence? Well, Fay Lee, also known as the Godmother of AI, just raised hundreds of millions of dollars to build an AI company around spatial intelligence. I'm going to explain what spatial intelligence is, and we're also going to watch an interview with her and famed Venture Capital firm a16z, where she talks about the future of artificial intelligence being one in which it understands the real world. And we're doing another giveaway of a 24in Dell monitor. You can win. All you have to do is subscribe to my newsletter. I'll drop all the information in the description below. So first, who is Fay Lee? She is a well-known computer scientist and has made many contributions to the world of artificial intelligence, and she really focuses and has a passion for visual intelligence, understanding the real world. Yan Laon said language alone is not enough to create a world model, which can also be described as artificial intelligence that understands the real world. It needs to actually see the world, and language alone can't do that.

that's why spatial intelligence is so promising. So let's look at some of her major contributions before diving into this interview of hers. So first, ImageNet. This is her most well-known contribution, and ImageNet is a large-scale visual dataset launched in 2009. It contains millions of labeled images across thousands of categories, which revolutionized the field of computer vision and deep learning. Basically, it is a dataset that helps computers see the real world. She was an assistant professor at the University of Illinois, she was an assistant professor at Princeton, and most recently became a full professor at Stanford University. But now she started her own company, and that's what she's going to talk about, along with what the company actually does in this interview. So here's the interview. It's on the a16z YouTube channel. I'll drop a link in the description below if you want to watch it in full, and I'm going to play this at 1 and A2 speed because it's quite a long video, so maybe walk through a little bit.

about how we got here, kind of like your key contributions and insights along the way. So, it is a very exciting moment, right? Just zooming back, AI is in a very exciting moment. I personally have been doing this for two decades plus, and you know, we have come out of the last AI winter. We have seen the birth of modern AI. Then we have seen deep learning taking off, showing us possibilities like plain tests. But then we're starting to see the deepening of the technology and the industry adoption of some of the earlier possibilities like language models. And now I think we're in the middle of a Cambrian explosion in almost a literal sense because now in addition to texts, you're seeing pixels, videos, audios, all coming out with Poss AI applications and the model. So, it's a very exciting moment. I know you both so well, and many people know you both so well because you're so prominent in the field, but not everybody grew up in AI, so maybe it's kind of

worth just going through like your quick backgrounds just to kind of level set the audience. Yeah, sure. So, I first got into AI at the end of my undergrad. I did math and computer science for undergrad at KCH. It was awesome. But then towards the end of that, there was this paper that came out that was at the time a very famous paper, the cat paper, from H. NE Lee and Andrew and others that were at Google Brain at the time. And that was like the first time that I came across this concept of deep learning. And to me, it just felt like this amazing technology. And that was the first time that I came across this recipe that would come to define the next more than a decade of my life, which is that you can get these amazingly powerful learning algorithms that are very generic, couple them with very large amounts of compute, couple them with very large amounts of data, and magic things started to happen when you comp pip those ingredients. So, I first came across that idea like around 2011.

2012-ish, and I just thought, like, Oh my God, this is going to be what I want to do. So, it was obvious you got to go to grad school to do this stuff. And then, um, sort of saw that FA was at Stanford, one of the few people in the world at the time who was kind of on that train. And that was just an amazing time to be in deep learning and computer vision specifically because that was really the era when this went from these first bits of technology that were just starting to work and really got developed and spread across a ton of different applications. And I remember during that time, around 2012, is when we really started to see the first commercially available image understanding tools. I remember seeing Meta launching a product like this where you could simply describe something in the image, and then it would circle it. And that, at the time, was absolutely mind-blowing. We probably take it for granted now, but really, when I saw that for the first time over a decade ago, I

was just completely blown away. So then, over that time, we saw the beginnings of language modeling. We saw the beginnings of discriminative computer vision. You could take pictures and understand what's in them in a lot of different ways. We also saw some of the early bits of what we would now call generative modeling, generating images, generating text. A lot of those core algorithmic pieces actually got figured out by the academic community during my PhD years. Like, there was a time I would just wake up every morning and check the new papers on archive and just be ready. It was like unwrapping presents on Christmas. I still feel that way anytime a new paper is published on archive and it's interesting, exciting. I can't wait to read it. I just love that he had that excitement so far in the past. But what was fundamental that my students and my lab realized probably earlier than most people is that if you let data drive models, you can unleash the kind of power that we haven't seen.

before and that was really the reason we went on a pretty crazy bet on imet, which is, you know, what just forget about any scale we're seeing now, which is thousands of data points at that point. NLP Community has their own data sets. I remember you see, and so essentially that is what OpenAI did. They took that famous paper Attention is All You Need, and they just continued to scale up the data set, the billions of parameters available to train the models. And what they figured out pretty quickly, probably before most other people, at least bringing it to a commercial application, is yeah, if we scale it up, those scaling laws definitely apply. The Irvine data set or some data set in NLP was small comparison. The community has their data sets, but all in the order of thousands or tens of thousands. We're like, we need to drive it to internet scale. And luckily, it was also the coming of age of the Internet, so we were riding that wave. And that's when I came to Stanford. So these epochs are what we

often talk about like imet is clearly the epoch that created you know or at least like maybe made like popular and viable computer vision in the Gen wave. We talk about two kind of core unlocks. One is like the Transformers paper, which is attention. We talk about stable diffusion. Is that a fair way to think about this, which is like there's these two algorithmic unlocks that came from Academia or Google, and like that's where everything comes from, or has it been more deliberate? So he's describing the attention is all you need paper, which is the basis for the Transformers model, which is the basis for all large language models today basically, and stable diffusion, which is the basis for creating all of the generative art models out there. And the interesting thing is something that just happened, or really we just figured out how powerful it can be, is inference time compute, and that is just another dimension that we're able to scale Artificial Intelligence on. So not only increasing the data that goes into training the models, but also just allowing the models

to quote unquote think over time and allowing them to think more and scaling up that thinking process. The number of tokens that they use will improve the output. So, we're in just as fascinating of a time as it was way back then. Or have there been other kind of big unlocks that kind of brought us here that we don't talk as much about? Yeah, I think the big unlock is compute. Like, I know the story of AI is often the story of compute, but even no matter how much people talk about it, I think people underestimated, right? And the amount of growth that we've seen in computational power over the last decade is astounding. The first paper that's really credited with the breakthrough moment in computer vision for deep learning was AlexNet, which was a 2012 paper where a deep neural network did really well on the ImageNet challenge and just blew away all the other algorithms that had been working on the types of algorithms that they had been working on more in grad.

school that Alex net was a 60 million parameter deep neural network. Um, and it was trained for 6 days on two GTX 580s, which was the top consumer card at the time, which came out in 2010. Um, so I was looking at some numbers last night just to, you know, put these in perspective. And funnily enough, the GTX 580s that they're describing, I right around that time, 2010-2011, purchased four or five of them, threw them into a laundry basket, put a fan next to it, and mined Bitcoin. The newest, latest, and greatest from Nvidia is the GB200. Um, do either of you want to guess how much raw compute factor we have between the GTX 580 and the GB200? Shoot, no, what? Go for it. It's uh, it's in the thousands. So I ran the numbers last night like that twoe r, that two we training run that of Six Days on 2 GTX 580s. If you scale seconds, it comes out to just under five minutes on a

single GB on a single gb200. Justin is making a really good point about the 2012 Alex paper. Yeah, and what they're describing is the exact reason why Nvidia is now one of the most valuable companies in the entire world. They've been making GPUs, parallel processing units, for decades, and it has only been used for video games and then for Bitcoin mining because the parallel processing and the kind of churning through all of that math is what the GPUs are really good at. And now, Jensen Huang and Nvidia had the foresight to realize the AI wave is going to need a ton of parallel compute, and they positioned incredibly well, maybe one of the best corporate success stories of all time. Practically the only difference between AlexNet and the ConvNet, what's the difference? Is the GPUs, the two GPUs, and the deluge of data. Yeah, well, so that's where I was going to go, which is like, so I think most people now are familiar with like quote the bitter lesson, and the bitter lesson says is if you make an algorithm, they'll be cute.

yeah just make sure you can take advantage of available compute because the available compute will show up right and so like you just like need to like why like on the other hand there's another narrative um which seems to me to be like just as credible which is like it's actually new data sources that unlock deep learning right like imag that is a great example but like a lot of people like self-attention is great from Transformers but they'll also say this is a way you can exploit human labeling of data because like it's the humans that put the structure in the sentences and if you look at clip they say well like we're using the internet to like actually like have humans use the alt tag to label images right and so like that's a story of data that's not a story of compute and so is it just is the answer just both or is like one more than the other or I think it's both but you're hitting another really good point so I think there's actually two epochs.

that to me feels quite distinct in the algorithmics here. So, like the imet era is actually the era of supervised learning. Um, so in the era of supervised learning, you have a lot of data, but you don't know how to use data on its own. Like the expectation of imet and other data sets of that time period was that we're going to get a lot of images, but we need people to label everyone and all of the training data that we're going to train on. Like a person, a human labeler, has looked at every one and said something about that image. Um, and the big algorithmic unlocks, we know how to train on things that don't require human-labeled data. So, I'm going to pause it there for a second. We've been talking about this over the last week in pretty recent videos. Humans are really the fundamental limiter of AI growth. Whether you're talking about the ore data sets or the labeling of the data sets, humans are the limitation or even the research itself. If AI models are able to generate

really great unlimited data for other models in a really high-quality way. Then, all of a sudden, we're going to have that intelligence explosion. Then we have unsupervised learning, which is basically what AlphaGo was all about. And so you have this system in which it could just try a bunch of permutations of the game Go and figure out which one worked best with essentially no human intervention whatsoever. And then finally, we have projects like AI scientists by S AI, which is actually doing the research on algorithmic unlocks for future models, and again, completely autonomously, with no human intervention. These multiple different dimensions are going to allow us to scale AI so much more quickly than we ever thought was possible. Today's video is brought to you by Mamut. Mamut AI brings all of the best models together in one place for one price: Claude, Llama, GPT-40, MRAW, Gemini Pro, and even GP01. And rather than having to pay for each of these AIs separately, you pay $10 to Mamut, and they bring it all together in one place. Plus, they have image

generation mid Journey flux Pro Dolly and stable diffusion again all for $10. Models are frequently updated as soon as they're released, so be sure to check out mamut for access to all the best models for one low price m.ai. That is m a MM o u t. A thanks again to mut. So let's walk to gen um. So when I was doing my PhD before that um. You came. So I took U machine learning from anding and then I took like beijan something very complicated from Dey CER. And it was very complicated for me. A lot of that was just predictive modeling y um. And then like I remember the whole kind of vision stuff that you unlock. But then the generative stuff has shown up like I would say in the last four years, which is to me very different. Like you're not identifying, you're not you know predicting something, you're generating something. And so maybe kind of walk through like yeah that's interesting that he says you're not predicting.

something, but that is kind of what it does. It's basically guessing at what the next token is, so it really is just predicting the key unlocks that got us there, and then why it's different, and if we should think about it differently, and is it part of a continuum, is it not. It is so interesting. Even during my graduate time, generative model was there. We wanted to do generation. Nobody remembers, even with the letters and numbers, we were trying to do some, you know, Jeff Hinton has had generated papers. We were thinking about how to generate, and in fact, if you do have, if you think from a probability distribution point of view, you can mathematically generate. It's just nothing we generate would ever impress anybody, right? So this concept of generation mathematically, theoretically, is there, but nothing worked. So then I do want to call out Justin PhD, and Justin was saying that he got enamored by deep learning, so he came to my lab. Justin's PhD, his entire PhD is a story, almost a mini story of the trajectory of the of

the uh field. He started his first project in data. I forced him to. He didn't like it in R. I learned a lot of really useful things. I'm glad you say that now. So, we moved Justin to um to deep learning, and the core problem there was taking images and generating words. Well, actually, it was even about there. I think there were three discreet phases here on this trajectory. So, the first one was actually matching images and words, like we have. We have have and we how much they my first both of my PhD and like ever my first academic ever was the IM retrial with scphs. And then we went into the Genera uh taking pixels generating words, and Justin and Andre uh really worked on that. But that was still a very, very lossy way of generating and getting information out of the pixel world. And then in the middle, Justin went off and did a very famous piece of work, and it was the first time that uh someone made it real-time, right? Yeah, yeah. So, the story there is there was

this paper that came out in 2015, a neural algorithm of artistic style led by Liam G. and it was like the paper came out and they showed like these real-world photographs that they had converted into Van Gogh style. And like, we are kind of used to seeing things like this in 2024, but this was in 2015. So, this paper just popped up on archive one day and it like blew my mind. Like, I just got this like gen brainworm like in my brain in like 2015 and it like did something to me. And I thought like, Oh my God, I need to understand this algorithm. I need to play with it. I need to make my own images into Van Gogh. So, then I like read the paper and over a long weekend, I reimplemented the thing and got it to work. It was very, actually very simple algorithm. Um, so like my implementation was like 300 lines of Lua because at the time it was pre-it was Lua. There was, there was um, this was pre-Torch. So, we

were using Lua torch, um, but it was like a very simple algorithm, but it was slow, right? So, it was an optimization-based thing. Every image you want to generate, you need to run this optimization loop, run this gradient descent loop for every image that you generate. The images are beautiful, but I just wanted to be faster, and Justin just did it. And it was actually, I think, your first taste of academic work having an industry impact. A bunch of people had seen this artistic style transfer stuff at the time, and me and a couple others at the same time came up with different ways to speed this up, yeah, um, but mine was the one that got a lot of traction, right? So, I was very proud of Justin, but there's one more thing I was very proud of Justin to connect to gen AI, is that before the world understood gen AI, Justin's last piece of work in PhD, which I knew about it because I was forcing you to do it, that one was actually inputting language and

getting a whole picture out. It's one of the first-gen work. It's using gang, which was so hard to use, but the problem is that we are not ready to use a natural piece of language. So, Justin, you heard he worked on Sing Graph. So, we have to input a graph language structure. So, you know, the sheep, the grass, the sky in a graph way. It literally was one of our photos, right? And then he and another very good Master student, Grim, they got that again to work. So, you can see from data to matching to style transfer to generative images. We're starting to see you ask if this is an abrupt change for people like us. It's already happening in a continuum, but for the world, it was. It's more the results are more abrupt. So, yeah, and it's interesting because a lot of people say the same thing today about AGI and ASI. It's not going to be one sudden point in time. It's going to be gradual and incremental. So, it's interesting to see.

that people who have been in the industry a long time, not myself, but Fay and Justin, they have seen this gradual continuum over a decade, multiple decades. Next, they are going to start talking about the importance of AI being able to understand the 3D, the natural world, and why that is going to unlock so much potential. So, let's watch. Seems for a long time like a lot of you, and I'll talk to you fa, like a lot of your research has been, you know, and your direction has been towards kind of spatial stuff and pixel stuff and intelligence. And now you're doing World Labs, and it's around spatial intelligence. And so, maybe talk through, like, you know, is this been part of a long journey for you? Like, why did you decide to do it now? Is it a technical unlock? Is it a personal unlock? Just kind of, like, move us from that kind of Meo of AI research to World Labs. Sure, for me, it is both personal and intellectual, right? My entire, you talk about my book, my entire intellectual

journey is really this passion to seek North Stars, but also believing that those North Stars are critically important for the advancement of our field. So, at the beginning, I remembered after graduate school, I thought my North Star was telling stories of images, because for me, that's such an important piece of visual intelligence as part of what you call AI or AGI. But when Justin and Andre did that, I was like, Oh my God, that's that was my live stream. What do I do next? So, it came a lot faster. I thought it would take a hundred years to do that. So, um, but visual intelligence is my passion because I do believe for every intelligent being like people or robots or some other form, knowing how to see the world, reason about it, interact in it, whether you're navigating or manipulating or making things, you can even build civilization upon it. Visual-spatial intelligence is so fundamental. It's as fundamental as language, possibly more ancient and more fundamental in certain ways. So, so it's very natural for me that um world

Labs is norstar is to unlock spatial intelligence and thinking back to what I said at the beginning of the video. This is kind of what Yan Lon has been saying. Language models alone are not enough to create World models, and we'll see if that's true or false. Faay seems to agree, although she's not saying it as definitively as Yan Leon has, but what she is saying is AI's ability to interpret the real world is absolutely fundamental. Let's keep watching the moment to me is right to do it like Justin was saying compute. We've got these ingredients, we've got compute, we've got a much deeper understanding of data, way deeper than image that days, you know, compared to those days. We're so much more sophisticated, and we've got some advancement of algorithms, including co-founders in Worldl like Ben Milen Hall and Kristoff Lar. They were at The Cutting Edge of nerve that we are in the right moment to really make a bet and to focus and just unlock that, so I

just want to clarify for folks that are listening to this, which is, so you know, you're starting this company, World Lab Spatial Intelligence, is kind of how you're generally describing the problem you're solving. Can you maybe try to crisply describe what that means? Yeah, so spatial intelligence is about the machine's ability to perceive, reason, and act in 3D and 3D space and time, to understand how objects and events are positioned in 3D space and time, how interactions in the world can affect those 3D positions, 3D 4D positions over space-time, and both sort of perceive, reason about, generate, interact with, really take the machine out of the mainframe or out of the data center, and putting it out into the world and understanding the 3D 4D world with all of its richness. So, a company that has a tremendous amount of 3D real-world data that can be used to train spatial intelligence is Tesla, of course. They have millions and millions of miles of

real-world data that has been ingested through the cameras in Tesla vehicles and just piped into a large database, and they are just training on it constantly. But they can do so much more with that data than just autopilot, which by the way is an incredible feat in itself. But that data can also be used to train Optimus, their robot. And the Optimus robot will then know how to operate within the real world, how to interpret the real world, understand the events that are occurring. Exactly what Justin is saying. Let's keep watching. Be very clear. Are we talking about the physical world or are we just talking about an abstract notion of world? I think it can be both. I think it can be both, and that encompasses our vision long-term. Even if you're generating worlds, even if you're generating content, doing that in positioned in free with 3D has a lot of benefits. Or if you're recognizing the real world, being able to put 3D understanding into the real world as well is part of it. So he's

saying not only can we interpret and understand the real world using spatial intelligence, but we can actually generate a world, and then we're getting into simulation theory and stuff we've talked about quite a bit on this channel, which I find fascinating. But then we think about things like Sora, which was able to generate video that looked incredibly realistic. The physics in the video looked realistic, and they weren't really using spatial intelligence to do that. So, there are multiple tracks happening at the same time right now that are trying to unlock this real-world intelligence. So, I mean, just for everybody listening, like the two other co-founders, Ben Hall and Christof Fler, are absolute legends in the field, at the same level. These four decided to come out and do this company now. And so, I'm trying to get dig to like, like why now is the right time. Yeah, I mean, this is again part of a longer evolution for me, but like really after PhD when I was really wanting to develop

into my own independent researcher both at for my later career. I was just thinking what are the big problems in AI and computer vision um and the conclusion that I came to about that time was that the previous decade had mostly been about understanding data that already exists um but the next decade was going to be about understanding new data. And if we think about that, the data that already exists was all of the images and videos that maybe existed on the web already and the next decade was going to be about understanding new data right like people are people have smartphones smartphones are collecting cameras those cameras have new sensors those cameras are positioned in the 3D world. It's not just you're going to get a bag of pixels from the internet and know nothing about it and try to say if it's a cat or a dog. We want to treat these treat images as universal sensors to the physical world and how can we use that to understand the 3D and 4D structure of the world um either in physical spaces or or or

generative spaces, so I made a pretty big pivot post-PhD into 3D computer vision, predicting 3D shapes of objects with some of my colleagues at the time. Then later, I got really enamored by this idea of learning 3D structure through 2D, right? Because we talk about data a lot, it's, um, you know, 3D data is hard to get on its own, um, but there, because there's a very strong mathematical connection here, um, our 2D images are projections of a 3D, and there's a lot of mathematical structure here we can take advantage of. So even if you have a lot of 2D data, there's, there's a lot of people done amazing work to figure out how can you back out the 3D structure of the world from large quantities of 2D observations. Oh, that's so interesting, something I had not really thought about. Yeah, we collect a ton of 2D data, 2D because it needs to basically be

projected onto a 2D screen, and so all of our cameras taking photos, taking videos, it's all 2D representations of 3D environments. Now, I immediately think to the Apple Vision Pro, and now I have an iPhone 16, and that can take spatial video, which is 3D to the best of my understanding. So, all of a sudden, we're going to have this huge flood of 3D video coming in to train these future models on, which is interesting to think about. I haven't thought a ton about it, so that's all I'm going to say about it for now, but let me know what you think in the comments. And then in 2020, you asked about breakthrough moments. There was a really big breakthrough moment from our co-founder Ben Milal at the time with his paper Nerf, um, n-radians fields, and that was a very simple, very clear way of backing out 3D structure from 2D observations that just lit a fire under this whole space of 3D.

computer vision. I think there's another aspect here that maybe people outside the field don't quite understand, as that was also a time when large language models were starting to take off. So, a lot of the stuff with language modeling actually had gotten developed in academia. Even during my PhD, I did some sly work with Andre K. P. A. on language modeling in 2014, LSTM. I remember LSTM RNs brus. Like, this was pre-Transformer. Um, but uh, then at some point, like around the GPT-2 time, like you couldn't really do those kind of model any in academia because they took way more resourcing. But there was one interesting thing, the Nerf, the Nerf approach that Ben came up with. Like, you could train these in an hour, a couple hours on a single GPU. So, I think at that time, like, this is a, there was a dynamic here that happened, which is that I think a lot of academic researchers ended up focusing a lot of these problems because there was core algorithmic stuff to figure out and

because you could actually do a lot with without a ton of compute, and you could get state-of-the-art results on a single GPU because of those dynamics. Um, there was a lot of research, a lot of researchers in academia were moving to think about what are the core algorithmic ways that we can advance this area as well. Uh, then I ended up chatting with Amore, and I realized that we were actually... She's very convincing. She's very convincing. Well, there that, but like, you know, you talking about trying to figure out your own independent research trajectory from your adviser. Well, it turns out we'd end up... Oh no, kind of concluding, converging on similar things. Okay, well, from my end, I want to talk to the smartest person. I call Dust, and there's no question about it. I do want to talk about a very interesting technical um, uh, issue or technical story of pixels that most people work in language don't realize is that pre-era in the field of computer vision. Those of us who work on

pixels. We actually have a long history in an area of research called reconstruction, 3D reconstruction, which dates back to the 70s. You know, you can take photos because humans have two eyes, right? So, in general, it starts with stereo photos, and then you try to triangulate the geometry and make a 3D shape out of it. It is a really, really hard problem to this day. It's not fundamentally solved because there's correspondence and all that. And then, so this whole field, which is older thinking about 3D, has been going around, and it has been making really good progress. But when Nerf happened, when Nerf happened in the context of generative methods, in the context of diffusion models, suddenly reconstruction and generation started to really emerge in now, like, within a really short period of time in the field of computer vision. It's hard to talk about reconstruction versus generation. We suddenly have a moment where if we see something, or if we imagine something, both can converge towards generating it.

and that's just to me a really important moment for computer vision, but most people miss it because we're not talking about it as much as LLMs, right? So, in pixel space, there's reconstruction where you reconstruct a scene that's real, and then if you don't see the scene, you use generative techniques, right? So, these things are kind of very similar, and we've seen a ton of examples of Nerf, Boal, Sidhu posts, Nerf examples all the time. They are incredible to see. Here's an image, a 2D image, that was translated into 3D, and this is a lot of the same technology that is being used in Apple Photos, and when you're scrolling through Apple TV and you're seeing the poster for a movie kind of rotate in 3D, this is all the same technology. Now they're going to talk about the difference between language models and spatial models, so let's watch throughout this entire conversation. You're talking about languages, and you're talking about pixels, so maybe

It's a good time to talk about how spatial intelligence and what you're working on contrasts with language approaches, which of course are very popular now. Is it complementary? Is it orthogonal? Yeah, I think they're complementary. I don't mean to be too leading here. Like, maybe just contrast them. Like, everybody says, Listen, I know OpenAI, and I know GPT, and I know multimodal models. And a lot of what you're talking about is like, They've got pixels, and they've got languages. And like, Doesn't this kind of do what we want to do with spatial reasoning? Yeah. So, I think to do that, you need to open up the black box a little bit of how these systems work under the hood. So, with language models and the multimodal language models that we're seeing nowadays, their underlying representation under the hood is a one-dimensional representation. We talk about context lengths, we talk about transformers, we talk about sequences, attention. Attention fundamentally, their representation of the

world is one-dimensional, so these things fundamentally operate on a one-dimensional sequence of tokens. So, this is a very natural representation when you're talking about language because written text is a one-dimensional sequence of discrete letters. So, that kind of underlying representation is the thing that led to LLMs and now the multimodal LLMs that we're seeing now. You kind of end up shoehorning the other modalities into this underlying representation of a 1D sequence of tokens. Um, now when we move to spatial intelligence, it's kind of going the other way where we're saying that the three-dimensional nature of the world should be front and center in the representation. So, at an algorithmic perspective, that opens up the door for us to process data in different ways, to get different kinds of outputs out of it, um, and to tackle slightly different problems. So, even at a coarse level, you kind of look at outside and you say, Oh, multimodal LLMs can look at images too. Well, they can, but I think that it's

They don't have that fundamental 3D representation at the heart of their approaches. That's so fascinating, and I'm learning as we're watching this right now. So, as we're adding multiple modalities to large language models, as he said, we're fitting three dimensions into a one-dimensional space, and that seems really inefficient now that I'm hearing him describe it that way. But if we take the opposite approach and we start with 3D, and that is fundamentally how the model works and understands the world is 3D, then converting it to 1D to use language might be just easy. But let's keep watching. I totally agree with Justin. I think talking about the 1D versus fundamentally 3D representation is one of the most core differentiations. The other thing, it's a slightly philosophical, but it's really important to, for me at least, is language is fundamentally a purely generated signal. There's no language out there. You don't go out in the nature, and there's words written in the sky for you. So, this again

sounds very similar to how Yan Laon describes large language models and their limitations. So, it seems like they're on the same page, although approaching solving the problem in different ways. Whatever data you're feeding, you pretty much can just somehow regurgitate with enough generalizability the same data out, and that's language to language. But the 3D world is not there. There is a 3D world out there that follows laws of physics, that has its own structures due to materials, and many other things. To fundamentally back that information out and be able to represent it and be able to generate it is just fundamentally quite a different problem. We will be borrowing similar ideas or useful ideas from language and LLMs, but this is fundamentally, philosophically to me, a different problem, right? So, language is 1D and probably a bad representation of the physical world because it's been generated by humans, and it's probably lossy. There's a whole another modality of generative AI models which are pixels.

and these are 2D images and 2D videos. And like one could say that like if you look at a video, it looks, you know, you can see 3D stuff because like you can pan a camera or whatever it is. And so like how would like spatial intelligence be different than say 2D video here? When I think about this, it's useful to disentangle two things. Um, one is the underlying representation and then two is kind of the user-facing affordances that you have. Um, and here's where you can get sometimes confused because um, fundamentally we see 2D, right? Like our retinas are 2D structures in our bodies and we've got two of them. So like fundamentally our visual system perceives 2D images. Um, but the problem is that depending on what representation you use, there could be different affordances that are more natural or less natural. So even if you are at the end of the day, you might be seeing a 2D image or a 2D video. Um, your

brain is perceiving that as a projection of a 3D world. So, there are things you might want to do, like move objects around, move the camera around. In principle, you might be able to do these with a purely 2D representation and model, but it's just not a fit to the problems that you're asking the model to do, right? Like modeling the 2D projections of a dynamic 3D world is a function that probably can be modeled. But by putting a 3D representation into the heart of a model, there's just going to be a better fit between the kind of representation that the model is working on and the kind of tasks that you want that model to do. So, our bet is that by threading a little bit more 3D representation under the hood, that'll enable better affordances for users. And this also goes back to the North Star for me. You know, why is it spatial intelligence? Why is it not flat pixel intelligence? It's because I think the arc

of intelligence has to go to what Justin cause affordances and uh and the Arc of intelligence. If you look at Evolution, right, the Arc of intelligence eventually enables animals and humans, especially humans as intelligent animals, to move around the world, interact with it, create civilization, create life, create a piece of Sandwich, whatever you do in this 3D World, and translating that into a piece of technology that three native 3D is fundamentally important for the floodgate um of possible applications. Even if some of them, the serving of them looks 2D, but it's innately 3D. And again, I want to think back to the Apple Vision Pro and the Oculus and this whole AR/VR Revolution that has yet to come. But all of a sudden, we are capturing all of this information about the 3D World, which can be used to train models that are based on spatial intelligence, spatial awareness. So very interesting to think about. I don't know a ton about these topics, but definitely fascinating to Think Through. I think

This is actually a very subtle and incredibly critical point, and so I think it's worth digging into. A way to do this is by talking about use cases. Just to level set this, we're talking about generating a technology, let's call it a model, that can do spatial intelligence. So, maybe in the abstract, what might that look like? Kind of a little bit more concretely, what would be the potential use cases that you could apply this to? So, I think there's a couple of different kinds of we imagine these spatially intelligent models able to do over time. One that I'm really excited about is World Generation. We're all used to something like a text-image generator, starting to see text-video generators where you put an image, put in a video, and out pops an amazing image or an amazing two-c clip. But I think you could imagine leveling this up and getting 3D worlds out. So, one thing that we could imagine spatial intelligence helping us with in the future is upleveling these

experiences into 3D, where we're not getting just an image out or just a clip out, but you're getting out a full simulated but vibrant and interactive 3D world. Yeah, we're definitely in the territory of simulation theory now. And once again, I'm going to bring up Apple Vision Pro and other AR/VR technologies. And again, thinking about Sora as a world generator. We've talked about world generators in the past, and even the AI Doom project that we talked about a couple of weeks ago, where a diffusion model was actually able to predict frame by frame the world of the 1990s game Doom with all of the logic, all of the 3D of it, but with no game engine, no predefined code, nothing like that. And that is what he's describing. This could transform video games, could transform all content, movies, TV. But think even beyond that. I mean, this could completely change the way that we view reality. Maybe for gaming, right? Maybe for gaming, maybe for virtual photography, like you name it. There's, I think, even if you got this to

work, there'd be a million applications for education. Yeah, for education. I mean, I guess one of my things is that like we, in some sense, this enables a new form of media, right? Because we already have the ability to create virtual interactive worlds, but it costs hundreds, hundreds of millions of dollars and a ton of development time. And as a result, like, what are the places that people drive this technological ability? It is video games, right? Because if we do have the ability as a society to create amazingly detailed virtual interactive worlds that give you amazing experiences, but because it takes so much labor to do so, then the only economically viable use of that technology in its form today is games that can be sold for $70 a piece to millions and millions of people to recoup the investment. If we had the ability to create these same virtual interactive vibrant 3D worlds, you could see a lot of other applications of this, right? Because if you bring down that cost of producing that kind of

content then people are going to use it for other things. What if you could have an inter-like sort of a personalized 3D experience that's as good, as rich, as detailed as one of these AAA video games that cost hundreds of millions of dollars to produce, but it could be catered to like this very niche thing that only maybe a couple of people would want, that particular thing. That's not a particular product or a particular roadmap, but I think that's a vision of a new kind of media that would be enabled by spatial intelligence in the generative realms. Yeah, agreed completely. That sounds incredible. I just want to describe a world, and I want to go explore it. It doesn't have to be a video game; maybe it is, maybe it isn't, but I just want to describe different worlds and see what it would be like to live within them, and that is just so cool, so futuristic. I want to think more about it, and I want to experience it. All right, in this last section, they're going to talk

about how all of this applies to AR and VR. Something that I've been talking about a lot in this video, so I'm excited to hear what they have to say about it. The case that Justin was talking about would be like the generation of a virtual world for any number of use cases. One that you're just alluding to would be more of an augmented reality, right? Yes, just around the time World Lab was being formed. Vision Pro was released by Apple, and they used the word spatial computing. We're almost like they almost stole our term, but we're spatial intelligence. So, spatial computing needs spatial intelligence. That's exactly right. So, we don't know what hardware form it will take. It'll be goggles, glasses, contact lenses, but that interface between the true real world and what you can do on top of it, whether it's to help you, to augment your capability to work on a piece of machinery and fix your car, even if you are not a trained mechanic, or to just be in a Pokémon Go Plus+ for

entertainment. Suddenly, this piece of technology is going to be the operating system, basically, for AR/VR. Mixr in. Yeah, this is definitely a new way to think about computing in general. As large language models get better, they seemingly are becoming the operating system of the future. But then beyond that, maybe spatial intelligence and spatial computing is the operating system of the 3D world's future. And there's so much to think about. It is so cool. Drop all of your thoughts in the comments below. I want to read them. I want to know what you think the limit, like what does an AR device need to do? It's this thing that's always on, it's with you, it's looking out into the world, so it needs to understand the stuff that you're seeing and maybe help you out with tasks in your daily life. But I'm also really excited about this blend between virtual and physical that becomes really critical. If you have the ability to understand what's around you in real time in perfect 3D, then it

actually starts to deprecate large parts of the real world as well. Like right now, how many differently sized screens do we all own for different use cases? Too many, right? You've got your phone, you've got your iPad, you've got your computer monitor, you've got your TV, like you've got your watch. Like these are all basically different-sized screens because they need to present information to you in different contexts and different positions. But if you've got the ability to seamlessly blend virtual content with the physical world, it kind of deprecates the need for all of those. It just ideally seamlessly blends information that you need to know in the moment with the right way mechanism of giving you that information. Yeah, I just talked about this in a previous video. The perfect implementation of hardware for AI in general is one that you basically don't have to wear at all. It just understands the world around you. It can see it, it can hear it, it can sense the 3D aspects of it, and then of course it

can project onto it. I don't know what that looks like. Mark Zuckerberg thinks it's glasses. Apple and Tim Cook think it's goggles. We'll see another huge case of being able to blend the digital virtual world with the 3D physical world is for any agents to be able to do things in the physical world. And if humans use this mixed art devices to do things, like I said, I don't know how to fix a car, but if I have to, I put on this goggle or glass and suddenly I'm guided to do that. But there are other types of agents, namely robots, any kind of robots, not just humanoid, and their interface by definition is the 3D world, but their compute, their brain by definition is the digital world. So what connects that from the learning to behaving between a robot brain to the real-world brain? It has to be spatial intelligence. So that's it for today. I cannot wait to learn more about this topic. It just seems so fascinating. Another approach to intelligence if you

enjoyed this video. Please consider giving a like and subscribe, and I'll see you in the next one.

What is Spatial AI? "The Next Frontier of AI Architecture"

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

What is Spatial AI? "The Next Frontier of AI Architecture"

What is Spatial AI? "The Next Frontier of AI Architecture"

Scripts