This is an artificial intelligence image generator. Given a text description of a picture, it will create, out of nothing, an image matching that description. As you can see, it is capable of generating high quality images of all kinds of different scenes.

And it’s not just images, in recent years generative AI models have been developed that can generate text, audio, code, and soon, videos too.

All of these models are based on the same underlying technology, namely deep neural networks.

In a few of my previous videos, I’ve explained how and why deep neural networks work so well. But I only explained how neural nets can solve prediction tasks. In a prediction task, the neural net is trained with a bunch of examples of inputs and their labels, and tries to predict what the label will be for a new input which it hasn’t seen before.

For example, if you trained a neural net on images labelled with the type of object appearing in each image, that neural net would learn to predict which object a human would say is in an image, even for new images which it hasn’t seen before.

Under the hood, the way that prediction tasks are solved is by converting the training dataset into a set of points in a space, and then fitting a curve through those points, so prediction tasks are also known as curve-fitting tasks.

And while prediction is certainly cool and very useful, it’s not generation. Right? This model is just fitting a curve to a set of points.

It can’t produce new images. So where does the creativity of these generative models come from, if neural nets can only do curve fitting?

Well, all of these generative models, are in fact just predictors. Yep, it turns out that the process of producing novel works of art can be reduced to a curve fitting exercise.

And in this video, you’ll learn exactly how. Suppose that we have a training dataset consisting of a bunch of images. We want to train a neural net to create new images which are similar in style to these training images.

The first thing you might try is to simply use the images as labels to train the predictor.

Here we don’t care about 
the mapping from inputs to outputs, so we can just use anything we like for the inputs, for 
example, a completely black image.

Predictors learn to map inputs to 
outputs according to their training data. So,   this predictor, once trained, should be able 
to map the dummy all-black image to new images, like those seen in the training set, right?

Err, 
okay, maybe not quite. That didn’t work so well.

Instead of producing a nice, beautiful 
picture, we just got this blurry mess. This demonstrates a very important fact about 
of predictors. If there are multiple possible labels for the same input, the predictor 
will learn to output the average of those   labels.

For traditional classification 
tasks, this isn’t really a problem, because the average of multiple class labels 
can still be a meaningful label.

For example,   this image could plausibly be given two 
different labels, both cat and dog would be valid labels.

In this case, a classifier would 
learn to output the average of those labels,   which means you end up with a score of 0.5 cat and 
0.5 dog, which is still a useful label. In fact, it’s arguably a better label than either of the 
original ones.

On the other hand, when you average   a bunch of images together, you do not get a 
meaningful image out, you just get a blurry mess.

Let’s try something a bit easier this time. How 
about, instead of generating a new image from   scratch, we try to complete an image which has 
a part of it missing. In fact, let’s make this really easy and suppose there is only one 
missing pixel, say, the bottom right pixel.

Can we train a neural net to predict the 
value of this one missing pixel? Well, as before, the neural net is going to 
output the average of plausible values   that the missing pixel can take.

But since 
it’s only one pixel that we’re predicting, the average value is still meaningful.

The average 
of a bunch of colors is just another color, 
there’s no blurring effect. So, this model works perfectly fine!

And we can use the value predicted by this neural net to complete images which are missing the bottom-right pixel. Great, so we can complete images with one missing pixel… What about two?

Well, we can do the same thing again, train another neural net on images with two missing pixels, using the value of the second missing pixel as the label. And then use this neural net to fill in the second missing pixel.

Now we have an image with just one missing pixel, and so we can use the first neural net to fill in that. Great.

And we can do this for every pixel in the image; train a neural net to predict the color of that pixel when it and all of the subsequent pixels are missing. Now we can “complete” an image starting from a fully black image, and filling in one pixel at a time.

Crucially, each neural net only predicts one pixel, and so there’s no blurring effect.

And there we have it, we have just generated a plausible image, out of nothing… There’s just one small problem. If we run this model again, it will generate exactly the same image… Not very creative, is it?

But not to worry, we can fix this by introducing a bit of random sampling. You see, all predictors actually output a probability distribution over possible labels.

Usually, we just take the label with the largest probability as the predicted value. But if we want diversity in our outputs, we can instead randomly sample a value from this probability distribution.

This way, each time the model is run, it will sample different values at each step, which therefore changes the prediction for subsequent steps, and we get a completely different image each time.

Now we have an interesting image generator.

But still, at the end of the day, this model is made of predictors. They take as input a partially masked image, and predict the value of the next pixel.

The only difference between this and a traditional image classifier is the label we used for training. The labels for our generator happen to be pixel colors which come from the original image itself, and not a human labeler.

This is a very important point in practice: it means we don’t need humans to manually label images for this model, we can just scrape unlabelled images off the internet.

But from the point of view of the neural net, it doesn’t know, nor does it care, that the label came from the original image.

As far as it’s concerned this is just a curve fitting exercise, like any other. The generative model we’ve just created is called an auto-regressor. We have a removal process, which removes pixels one at a time, and we train neural nets to undo this process, generating and adding back in pixels one at a time.

This is actually one of the oldest generative models, the very earliest use of auto-regression dates back to 1927, where it was used to model the timing of sunspots.

But auto-regressors are still in use today. Most notably, Chat-GPT is an auto-regressor.

Chat-GPT generates text by using a transformer classifier to output a probability distribution over possible next words, given a partial piece of text. However, auto-regressors are not used to generate images anymore.

And the reason is that, while they can generate very realistic images, they take too long to run.

In order to generate a sample with an auto-regressor, we need to evaluate a neural net once for every element. This is fine for generating a few thousand words to make a piece of text, but large images can have tens of millions of pixels.

How can we get away with fewer neural net evaluations?

For our auto-regressor, we removed one pixel at a time. But we don’t have to remove only one pixel, we could, for example, remove a 4 by 4 patch of pixels at a time.

And train the neural net to predict all 16 missing pixels at once.

This way, when we use our model to generate an image, it can produce 16 pixels per evaluation, and so generation is 16 times as fast.

But there is a limit to this. We can't generate too many pixels at the same time. In the extreme case, if we try to generate every pixel in the image at once, then we're back to the original problem: there are many possible labels that get averaged together into a blurry mess.

To be clear, the reason why the image quality degrades is that, when we predict a bunch of pixels at the same time, the model has to decide on the values for all of them at once.

There are lots of plausible ways that this missing patch could be filled in, and so the model outputs the average of those. The model isn't able to make sure that the generated values are consistent with each other.

In contrast, when we predict one pixel at a time, the model gets to see the previously generated pixels, and so the model can change its prediction for this pixel to make it consistent with what has already been generated.

This is why there's a trade-off, the more pixels we generate at once, the less computation we need to use, but the worse the quality of the generated images will be.

Although, this problem only arises if the values we are predicting are related to each other.

Suppose that the values were statistically independent of each other, that is, knowing one of them does not help to predict any others. In this case, the model doesn't need look at the previously generated values, since knowing what they were wouldn't change its prediction for the next value anyway.

In this case you can predict all of them at the same time without any loss in quality.

So, that means, ideally, we want our model to generate a set of pixels that are unrelated to each other.

For natural images, nearby pixels are the most strongly related, because they are usually part of the same object. Knowing the value of one pixel very often gives you a good idea of what color nearby pixels will be.

This means that removing pixels in contiguous chunks is actually the worst way to do it. Instead, we should be removing pixels that are far away from each other, and hence more likely to be unrelated.

So if in each step, we remove a random set of pixels, and predict values for those, then we can remove more pixels in each step for the same loss in image quality, compared to contiguous chunks.

In order to minimize the number of steps needed for generation, we want the pixels we remove in each step to be as spread out as possible.

Removing pixels in a random order is a pretty good way of maximizing the average spread, but there is an even better way.

We can think of our generative model as two processes: a removal process that gradually removes information from the input, until nothing is left. And a generation process that uses neural nets to undo the removal process, generating and adding back in information.

So far, we have been completely removing pixels. But rather than completely removing a pixel, we could instead remove only some of the information from a pixel, by, for example, adding a small amount of random noise to it.

This means we don’t know exactly what the original pixel value was, but we do know it was somewhere close to the noisy value.

Now, instead of removing a bunch of pixels in each step, we can add noise to the entire image.

This way, we can remove information from every pixel in the image in a single step, which is the most spread-out way of removing information. And since it's more spread out, you can remove more information in each step, for the same loss in generation quality.

There is one small problem with this though. When we want to generate a new image, we need to start the neural net off with some initial blank image.

When we were removing pixels, then every image eventually ends up as a completely black image, so of course that’s where we start the generation process from.

But now that we’re adding noise, the values just keep getting larger and larger, never converging to anything. So where do we start the generation process from?

We can avoid this problem by changing our noising step slightly, so that we first scale down the original value and then add the noise. This ensures that, when we repeat this noising step many times, information from the original image will disappear, and the result will be equivalent to a pure random sample from the noise distribution.

So we can start our generation process from any such noise sample. And there we have it, this is known as a denoising diffusion model. The overall form is identical to an auto-regressor, the only difference is the way in which we remove information at each step.

By adding noise, we can spread out the removal of information all across the image, which makes the predicted values as independent of each other as possible, allowing you to use fewer neural net evaluations.

Empirically, diffusion models can produce high-quality photo-realistic images in about a hundred steps, where auto-regressors would take millions.

Now that we understand how these generative models work at a conceptual level, if you are ever going to implement these models in practice, there are a few important technical details that you should be aware of.

First, in the procedure I described for auto-regression, I used a different neural net in each step of the process. This is certainly the best way to get the most accurate predictions, but it’s also very inefficient, since we need to train a whole bunch of different neural nets.

In practice, you would just use the same model to do every step. This gives slightly worse predictions, but the savings in computation time more than make up for it.

In order to train a single neural net to perform all of the generation steps, you would remove a random number of pixels from each input, and train the neural net to predict the corresponding next pixel of each input.

Additionally, you can also give the number of pixels removed as an input to the neural net, so that it knows which pixel it’s supposed to be generating.

Now this one neural net can be used for all generation steps.

In the setup I just described, for each training image, the neural is trained on only one generation step for that image. But ideally, we would like to train it on every generation step of every image, we can get more use out of our training data that way.

If you did this the naïve way, you would have to evaluate the neural net once for every generation step, which means a lot more computation.

Fortunately, there exist special neural net architectures, known as causal architectures, that allow you to train on all of these generation steps while only evaluating the neural net once.

There exist causal versions of all of the popular neural net architectures, such as causal convolutional neural nets and causal transformers.

Causal architectures actually give slightly worse predictions, but in practice, auto-regression is almost always done with causal architectures because the training is so much faster.

The generation process for causal architectures is still exactly the same though.

For diffusion models, you can’t use causal architectures, and so you do have to just train with each data point at a random generation step.

I described the diffusion model as predicting the slightly less noisy image from the previous step. However, it’s actually better to predict the original, completely clean image at every step.

The reason for this is it makes the job of the neural net easier. If you make it predict the noisy next step image, then the neural net needs to learn how to generate images at all different noise levels.

This means the model will waste some of its capacity learning to produce noisy versions of images. If you instead just have the neural net always predict the clean image, then the model only needs to learn how to generate clean images, which is all we care about.

You can then take the predicted clean image and reapply the noising process to it to get the next step of the generation process.

Except that when you predict the clean image then, at the early steps of the generation process, the model has only pure noise as input, so the original clean image could have been anything, and so you get a blurry mess again.

To avoid this, we can train the neural net to predict the noise which was added to the image.

Once we have a predicted value for the noise, we can plug it into this equation to get a prediction for the original clean image. So we are still predicting the original clean image, just in a roundabout way.

The advantage of doing it this way is that now the model output is uncertain at the later stages of the generation process, since any noise could have been added to the clean image.

So the model outputs the average of a bunch of different noise samples, which is still valid noise.

So far we’ve just been generating images from nothing, but most image generators actually allow you to provide a text prompt describing the image you want to make. The way that this works is exactly the same, you just give the neural net the text as an additional input at each step.

These models are trained on pairs of images and their corresponding text descriptions, usually scraped from image alt text tags found on the internet.

This ensures that the generated image is something for which the text prompt could plausibly be given as a description of that image.

In principle, you can condition generative models on anything, not just text, so long as you can find appropriate training data. For example, here is a generative model that is conditioned on sketches.

Finally, there’s a technique to make conditional diffusion models work better, called classifier free guidance.

Why Does Diffusion Work Better than Auto-Regression?

This is an educational channel for all things algorithmic, including but not limited to computer science, machine learning, physics, and mathematics.

Why Does Diffusion Work Better than Auto-Regression?

Why Does Diffusion Work Better than Auto-Regression?

Scripts