you might be thinking that these are some pretty cool or pretty decent Nerf-looking images um and you would be right. This is a Nerf, but it's actually a single image that generated these uh, which is what this paper proposes. So basically, you give it an image and it produces a 3D uh representation of that image. Uh, and this is the first model I've seen that actually does like a pretty good job with it. These images, these are pretty good. It's literally a single image and it generates all of this uh. Some of them have pretty good detail in there, yeah. Uh, there's some more down here, uh, can play around with this draff and it looks a bit weird but like the main shape is there. Uh, this one has a little more texture to it and there are some more down here that my tablet cannot load but yeah, this is the paper LRM large reconstruction model for single image to 3D. So one image produces a 3D representation of that image um to start, uh, just hop into it so this is the

model here. Basically, what they do is they collect this large data set of 3D objects. Well, they don't collect it, but there exists a lot of 3D object data sets. They list them all down here. Down here, and they list them all, but there's a lot of right here. So object OB obverse MV imag net some more. So you basically have this data set of 3D objects, and what you can do is you can basically pose a camera around those objects like a virtual camera to get a 2D representation of that image. And basically, you want to train your model so that it's given this 2D image here, and then it produces the 3D image that went with that object. So yeah, this is the model, and we'll just go through it step by step. It's pretty. It's a pretty cool idea how they do this. So yeah, let's just get in, do it. So the first thing is

how do you encode this image? So if we look right here, at the top, um, just yeah. So if we look here at the top, you can see that they have this part here. So this part here is just a Dino encoder. So if you know if you've heard of Dino, then, uh, you'll, you'll kind of know, um, how this works. But basically, Dino is a self-supervised model, um, basically trained in a way where, given some image, so you have some image, go with the cat again, like usual. So you have this image here, and you basically have this teacher model, and the teacher model is going to give you some encoding. So this is just going to be a really high-level review of Dino, because this, this isn't about, uh, Dino. They just, they just use Dino. I think it's just Dino V1. V2 came out, and Dino V2 is crazy good. Um, so you feed this through a teacher model, and this gives you some encoding, uh, and then, uh, you basically take a patch.

of this. So let's say we take this patch here. Uh, and so we have this little patch here, and it'll just be kind of the corner of the cat's face, uh, like that, maybe. Now you feed this through a student model, and then this spits out an encoding. We'll call this maybe subs and see e subt. Uh, you want these to be basically the same, so basically the student, given less information, has to kind of, uh, come up with a representation of like a good representation of this image, uh, to kind of match what the teacher has learned. But basically it's a self-supervised method, uh, that given an image, it gives you an encoding. That's the big point of it. But yeah, basically it's just, uh, self-supervised learning, uh, student-teacher, uh, and, um, yeah, that's all go into for that. But, uh, what this, yeah, so it gives you this output here, which is 32 by 32x768. So you give it this image 512x512x3. Uh

just throw that through Dino, and uh this is Dino. It's just um, it's just a vision Transformer, uh, and they use Transformers throughout this like you do. You always use a Transformer. So you give it this image, you encode it using a convolutional neuron Network, um, and you send it through. You send it through Dino, and it gives you your encoding out. Now, Dino is going to give you, uh, some, uh, encoding, just a class encoding, which is your embedding. So it'll give you out, uh, it'll give you a bunch of tokens because you basically patchy your image before you send it through. So before you send it through Dino, you basically patchy the, um, the image because it's vit, and, uh, you basically have these individual tokens. And the tokens are, uh, 32 by 32, or, um, no, the to, there's 32x32 tokens. That's the number of patches. And then there's, uh, they're each of Dimension 768, and these all represent encoded, uh, patches, but it gives you a class token.

and this class token is basically used for um this self-supervised approach up here, but they take all the encodings because you need a lot more like one encoding isn't enough to kind of represent this whole 2D image here enough so that you can reconstruct it and do a 3D image. So they take all the outputs of the dyo model um instead of just a class token, and that gives these embeddings right here. So basically we now have this 2D representation of our image um, which is 32 by 32 that's our sequence length um by 768 which is our feature size, and this is our 2D encoding, and we want to decode this to a 3D object. Now the second part is here, uh basically, uh I'll just show this over here so you start with your image throw it through dyo you get out your image encoding. Now what they do is they take these positional encodings so we want to en we want to decode this to some three-dimensional

space. I'll just go to the three-dimensional space. So our three-dimensional space is going to be represented by a tri-plane. So it'll be three planes. And if you basically take three points X, Y, X, Y, and Z, so given you can give me three points, and I can give you a point in this three-dimensional space based off off those three points. If we have perpendicular tri-planes, so the idea is that we just want our model to model these tri-planes here. So the output of our Transformer, the end goal is basically or the end goal of our Transformer is to output this 64x64x3. And basically each of these tri-planes will have 64 by 64 encodings to it. And you can think of the positional encodings as just like hopping along this plane here. So um, that one there and the first positional encoding may be this spot and then this spot, this spot, this spot, and so on. And then you can think you just concatenate that so that would be our um

our first plane and then the second plane will have the same idea. So you maybe we look at this plane and the positional encoding would just represent some ordering along this plane. So that's the idea. We want our model to model these triplanes. We don't want it to model this volumetric space because the volumetric space is expensive. We want it to model the triplanes because we just need three. Like a volumetric space would be 64 by 64 by 64. It's very very large but we can compress it if we 3x 64x 64. And this actually isn't going to be 64x 64 by 3. Or it's actually going to have an embedding Dimension as well. So our embedding Dimension. I think it's yeah it's 80. So each point on here isn't a point it's actually a 80 dimensional vector. And we'll get into how that's used later but just think of this part here. We want to model three triplanes.

and these three Tri-planes allow us to pick any point, any combination, linear combination of points here. So we can get any point in this volumetric space here. And this volumetric space is how we're going to model our resulting object. So right now we have our two-dimensional encodings. The way we're going to generate this is we give it these positional encodings. Um, here. So the positional encodings are just going to be 3 by 32x32. So this represents our Tri-plane. So you have three planes and the resolution of each plane is 32x32, which is expanded later in a convolution or not network. And we have an embedding for each point on those planes. So we have these positional encodings and this is basically our sequence length. So 3x32, 2 by 32, and a normal transformer. That's your sequence length here. And this is just your embedding dimension. Now you just send both of you send this through a transformer. Uh, this

is just your base uh what you're going to do self-attention with. And if you've looked at attention is all you need, you know that you can condition a sequence. So if we have this sequence here, we can condition it on another sequence via cross-attention. And that's all they do here. Um, they just do cross-attention using the 2D image on the positional encodings. And the idea is that as the Transformer kind of uh kind of changes the positional encodings as the as it's conditioned on the 2D image, it'll slowly get an understanding of how this 3D representation needs to look. Uh, so yeah, you just send it through these positional uh, you send the positional encodings uh through the Transformer conditioned on the 2D image, and you get out your Tri-plane encodings. Uh, you then expand the tri-plane encodings so it has a really large. Uh, this is 1024. So if we uh, it's in a way, it's kind of like um, a unit uh, except you only have

the top part of the unit uh with the unit, you have a lot of channels with the small resolution. At the top of the unit, you have um a large, very high resolution with few channels in it. So, basically, you compress the number of features you have. So here, you basically compress the number of features and expand the resolution. Uh, and you do this through what's called a it's a dcom or an inverted convolution. Uh, basically, you're just, you're, you're decompressing the feature space. You're decompressing the features into the resolution to get a higher resolution embedding. And that's shown here. So you start out with your positional encodings. We'll get into what that is in a sec. And uh, you have your cross attention. So, cross attention just takes this encoding here and it throws it through the attention module. Um, cross attention is formulated, if I remember correctly. Um, so we all know qkv queries, keys, and values. Uh, I think the keys are going to be your two-dimensional data.

your values are going to be your two-dimensional data, and then your queries are going to be your positional encodings. Um, but you can basically, uh, formulate this so that the shapes work out, and the output is the exact same shape as the positional encodings. Uh, kind of a hacky way of doing this. Um, it's how they do it in Stable to Fusion. So you basically have your matrices, you multiply them in the, in a, in a way so that the output is the same shape. It retains the shape, but it conditions it on the 2D image or in this, yeah, on on the 2D image encodings. Um, yeah. So if you ignore this, if you ignore these M's, you basically have normal, um, a normal decoder Network. So you have your cross-attention, you have your residual connection, you have your self-attention, you have residual, and then you have your MLP, and then another residual, and you just stack these, you just stack these layers over and over again. You get out your output, which is 3x 32x 32, uh, that's your

sequence length that's your embedding dimension uh which is the same as the input as a Transformer does and then you do your deconvolution where you basically uncompress your feature your features so you decrease the feature size size and you increase the resolution so the resolution basically doubles for each plane you go from 32 uh by 32 to uh 64x 64 and your your dimension your embedding dimension goes down from 1024 to 80 and yeah you can reshape this and you basically get your triplane which is 3x 64x 64 by 80 so that's our triplane. [Music] um I will get to this after the uh the last part of the model so last part of the model is so we basically have this threedimensional triplane and on any part of the triplane we can get this we can get these embeddings now the embeddings are kind of useless because it doesn't really like it's just some embedding but what they do is they use a

Nerf model now a Nerf model um BAS, basically a Nerf model given say XYZ, you throw this through a Nerf model. So given XYZ, uh you've probably seen a Nerf model before. It's basically, uh, the model itself. So this model here is like maybe a few linear layers, uh, at least for the original Nerf model, and then it'll output some information. I think it was like the camera orientation, uh, or maybe that was the input, but basically it would, it would output like the RGB, so r g b, um, and then the density, uh, yeah. And this, you, you give it your, uh, your camera angle, and I think you also give it the, the orientation, um, but you give it your, your camera info. So where is it kind of like looking at on the scene, um, and then you have this like 3D model here, maybe it's like some vas looking thing, uh, it looks like that. So your model represents your entire scene, and in this case it would be some 3D scene.

and you basically query the model for XYZ positions um and and the camera orientation, and it basically gives you kind of the image there. So this would be the output, and this would be the kind of hidden representation of the model um so yeah that that's kind of what what you're going to get with the Nerf model. So the Nerf model is you you get one pixel output for uh you basically query the model, and it gives you a pixel output. It'll make more sense in this in this context, I feel like uh since it's slightly different. So basically in this Nerf model, you give it XYZ so you query uh X, Y, and Z here, uh this is our X point, this is our y point, and this is our Z point, and remember each of these points is actually an embedding so uh we don't actually have three numbers, we actually have three vectors, uh and these are Point features, and basically we're saying what is what what does it look like, what does our output image look like at this

point here uh and in this case maybe it's like nothing so this will output um yeah so this outputs uh the RGB and density. RGB and density so at every Point here you get uh three values. You get the r, the G, the B value and the density. So maybe for this point here it gives you a density of zero meaning that it's just nothing so it would look like nothing but maybe if we look at this the origin here we give it the origin uh maybe that's this point here and it gives you uh B some yellow RGB combination with a high density so our MLP gives us that our MLP is just our Nerf model given XYZ coordinates so some coordinate on here you give it the embeddings representing that coordinate and it decodes those embeddings to the RGB value at that coordinate and that's our Nerf model uh neural Radiance field yeah RGB and uh density um yeah so that's what it gives us and then uh you can kind of render the image um using that using uh some camera pose.

at some point here um but that's the entire model. Uh, we'll go over this in a sec, but yeah, basically you give it a 2D image, you send it through dyo Dino, then outputs an embedding. You then send the positional encodings through this cross-attention decoder attention, uh, Network, and you condition it on your, uh, Doo embeddings. This will output some feature representation, some, uh, representation of your 3D space, which you then expand, uh, you increase the resolution of that and your resolution, uh. After you increase the resolution, you now have this Tri-plane representation. You can then send your Tri-plane representation through your Nerf model, which then gives you your output, uh. If you query the Nerf model for every point on here, it'll give you your output, um, 3D representation of the model or of the, of the image, um. This is a massive architecture, is, is really big. There's a lot of moving Parts, but the nice part is your loss function is right here, and that's all you have to model, uh, your.

loss function is just the Reconstruction of this here. It's just the Reconstruction law. So I said I would get to this. That's just because it's kind of it's it's it's not as much machine learning as it is um more I think I think it more has to do with like uh rendering in in video games and how you how you position the camera correctly. So it's not really machine learning. However, it is it is interesting because they they they mention that having this having the this information helps a lot. So basically let me go down here. They say they have they have these camera features so they construct the the camera features C in R20. So the idea is so we need these camera features we need to encode where is this image. So let's say we have let move this over. We have a space like this. This is our three-dimensional space x. That would be y. That would be Z. And this is just going to be our training data. So maybe we have some ugly

looking vase that looks like this like that that's a that's a vase. Uh maybe our vase has some fantastic beautiful flowers that look like this. Wow, that is that is a vase with flowers. So this is the 3D model that we kind of want of uh represent, and we want our model to learn from. So this is training data, and we now need to kind of get an image of this data. So let's say we kind of have a camera here, and our camera will give us some 2D uh projection of this threedimensional space. And let's say it kind of looks some, it has this uh, it has this field, it has this receptive field. So the receptive field picks up, you know, a camera, it picks up this three-dimensional shape, and it'll give you an output image. Now, the output image is actually going to be, it's not directly on the camera. So if this was like your, your camera here, your camera lens, then the output image is actually, there's, there's a focal length to it, so focal

length, and it's actually, it's not, it's not exactly where you would capture it, where you think you would capture it, um, and this gives you your, your output image after, um, processing and stuff like that. So, uh, let me draw this like that. So, this is some 2D shot, uh, if we have a camera right here, it looks at the scene and it basically grabs the, the 2D image there, and, uh, yeah, so that's our, the kind of the setup, and you can move this camera around anywhere you want because you have this 3D model. Now, now the idea is we want to give the model information on where the camera was, kind of what position was it in to inform the model, uh, instead of it just saying, Oh, I have an image, I have no idea where this is at, now I have to reconstruct the whole entire image from that. Uh, that would make it quite difficult for the model to learn, learn where this, uh, learn where the image kind of fits into the 3D model.

um so they give it a bunch of information. The main uh kind of piece of information is called um it was called the extrinsic uh camera. The extrinsic the extrinsic Matrix. So e is the extrinsic uh Matrix and the SE trans Matrix basically has kind of two two main parts to it. So let me just write out kind of the default form R22 R23 R 31 TX Ty y TZ and there 0 00 01. So this is the Matrix here and this upper left part here basic basally represents this is your camera rotation with respect to the axes. So uh basically with respect to these axes here where is kind of the camera. So you can model that uh using these here um oh and the what we're trying to model is we're trying to model so if this is our camera uh and this is our world we're basically trying to model a camera to world uh representation or transformation. Basically the model doesn't have to learn this transformation. It can instead use these features that are automatically created.

and that are used in in that are used in like video games and stuff. These, uh, these camera to World transformations to form these the a 3D object to kind of inform, uh, the model where the camera's at. Um, so this upper part here, this upper part is the camera rotation with respect to the world axes. This part here is basically the camera position in the world. So this is the position, this is the rotation, and then this last part down here is just to main homogeneity, uh, maintain that in the in the um whenever you do a linear transformation using this 4x4 Matrix. So this gives us 16 values. The other four values are mentioned over here. So there's four more values, there's x, uh, folk X, folk y. So, uh, that's just the focal length. And this is PPX, ppy. So that's this is the focal length, um, and PPX, ppy is basically just the the point, uh, kind of on the camera, uh, the what is they call it. They called it the principal point. And I'm definitely not

explaining this as well as it needs to be explained. This part here, as this is not where my knowledge resides. I don't know much about the um camera part, but if you are curious in this, uh, it's camera features. This is the 4x4 extrinsic Matrix. Um, it's basically just informing the model where is the camera, uh, according to our 3D model. So you move this camera around, where is it at, so that the model knows where the camera is, where the camera is at, uh, with respect to the 3D model, instead of just taking a random, uh, 2D picture and giving that to the model. So just more information, and the model can use that. Uh, if we go back up here, you put it in, uh, you put it right before every module, and this actually acts as a normalization. Uh, they model the normal, so it acts as a normalization, um, right here. Basically, you do a layer Norm, so this gives you your, uh, a layer normalization, basically zero. It gives you a zero mean and a one.

variant or uh yeah zero mean one variance, and then the model learns uh how to shift so uh how to how to shift that around so it'll has a learned mean, and then it has a learned variance, so you have a distribution like so. Your distribution is all out of whack. Maybe I'm going to represent it as a normal distribution, uh maybe it's very small, and it's all the way over here. You then zero mean, uh one variant, so you kind of shift it and you standardize it, and then the model can learn where to move this. So maybe it wants to move it over here, and maybe it wants to expand it a little bit larger, uh you then shift it by 1 plus gamma. And Gamma is just some linear combination of these features over here of the camera features, and then you shift it by the camera features as well. So basically you just normalize it, and then it learns a mean and variance according to the camera features, so it's just helping the model out a little bit, um yeah, so that's the entire

model um now the loss fun function uh is right here. The loss function is simple so like a simple loss function is amazing having a simple loss function uh you can't get much better than that. Uh usually whenever you have a model like this you have really complicated loss function but literally all they do is the mean squared error between the ground truth and the prediction for the pixel Recon or for for the pixels and output or I guess foxholes um and they do it uh they also have this other loss here called the LPS and basically this loss function is supposed to kind of uh model human perception a little bit um. You basically feed these both through some type of model so you feed the prediction through a model and you feed the uh the ground truth through a model and you basically do uh you compute some similarity maybe that's MSE between all the activations in the model so if I have I'll represent it as an image uh it makes it easier if I have some image here so this is going to be our ground truth image.

and then I have a reconstructed image. We'll call this X, and this will be X hat. You basically send this through the model. So the model mod is going to have a bunch of layers in it. Yeah, we draw like that. So this is the same model here. I'm just drawing it twice. So this is our model. It has some features, and maybe it gives you a class output, or maybe it predicts 01, something like that. Maybe this is some imagenet classifier. So it would actually have like 100 classes here, more. But the encoded images, you basically do MSSE for the activations between all of all of the layers for both these images. And the idea is that this basically is it matches human perception. So instead of it being like pixel reconstruction, a direct reconstruction, it's more of a visual reconstruction, at least hopefully assuming you have a good model. So this is just some model, maybe this is some imagenet resnet model, something like that. But that's your loss.

reconstruction loss, along with this extra loss. Um, basically make it more visually appealing, make the output more visually appealing in case the Reconstruction isn't perfect because it's not going to be and you do this for V total views, uh, around the image. So basically, you just take a ton of views, you give that to the model, and the model tries to reconstruct this, and that's your loss given those views, uh, now that is the entire model. They do some extra things, uh, they normalize it. So basically, the shape of the output Box is 1 by 1. So, um, that would be this box here. Uh, they just normalize everything because normalization is it just it helps the model. You don't have exploding values. Um, yeah, the rendered images are of resolution 1024x1024, and the camera poses are sampled from a ball radius 1.53 with the height of that. So if you're curious, they have all the training details in here. Uh, here are some examples, yeah. So training, they train this on 1208

Nvidia um a100 dpus, which is absolutely insane. Of course they do, of course they training on J 128 Nvidia gpus. That's quite quite a lot um, yeah. So that just means that you probably can't train this yourself uh, and it only takes three days, only three days to train if you got 100 gpus. Uh, and then inference, so inference uh, basically it takes an arbitrary image as input. So you give it some image uh, you oh so squared and background removed. So you have to remove the background. They don't model the background at all um. If we look at these views, it's just a white background. Um, they don't want to try modeling the background. I think that would make it quite difficult to try to model the background because it's very ambiguous um. And it assumes the unknown camera parameters to be the normalized camera that were applied to train the data. So basically whatever the default camera position was, they assume that that is the position of the given image that you have. And then they query 384 by

384 by 384 points from the reconstructed triplane Nerf and extract the mesh grid using marching cubes. So they basically query where this right here a bunch of times um to get the output mesh uh using the Nerf model. And they say the entire process only takes 5 seconds to compute uh if you have an a100 um. So it's not it's it's pretty fast to get this reconstruction prediction. Uh I don't think there's anything else besides that uh. Yeah features uh the output scores aren't like there's no nobody competing with this model so the output scores aren't that interesting. Uh at least and in my opinion yeah more layers the better so yeah. And then they have some more images cuz like they're the only ones who have really done this well on this task from what I know. So they don't really competing with anybody they they really have a good Baseline um. Yeah that's the model it's pretty pretty massive model you got viit uh you got this cross attention you got

this output Nerf model they're really throwing a ton of stuff at it, and it looks, it looks pretty good. It's looks like it looks like it's getting there. Um, the output meshes are pretty good, and uh, I thought, I thought this was a pretty cool paper, different from the normal llms and diffusion, diffusion papers. Um, I see all the time, so yeah, nice change of pace and looking forward to seeing how well this model actually performs. Like usual, I don't think they open sourced it, so we will see once somebody creates a, uh, or somebody recreates this paper, but it looks like it has promise.

LRM: Large Reconstruction Model for Single Image to 3D

Just some guy making exploring and making videos about current AI topics.

LRM: Large Reconstruction Model for Single Image to 3D

LRM: Large Reconstruction Model for Single Image to 3D

Scripts