Okay, good afternoon everyone. Welcome to the third lecture of efficientml.ai. Um, today we are going to jump into a very important topic about model compression, which is pruning and sparsity. Uh, we will have two lectures that cover pruning and sparsity. Uh, since this is a pretty important technique, and this will be the first lecture, so today's AI is too big. We have seen this figure a lot of compute to achieve high accuracy, and we want to bridge the gap between the supply and demand of efficient AI computing. So, uh, part one of this course will be about inhibition inference, and we'll have four major pillars. Okay, the first pillar is about pruning. Uh, we are going to send two lectures about that. Lab zero is already out, so flag one will be about booting. It will be out by um, next week. Okay, so keep an eye on the homework, and then we are going to have another two lectures about organization. Okay, for this today, this is next Tuesday, like we'll be out by Tuesday, and then the third part will be neural architecture.

search okay in your architecture search followed by uh knowledge distribution okay uh. We will have Lab at school uh which is about uh quantization okay on September 20th followed by um the third lab which will be related uh to neural architecture search which will be out at that time and after uh the student break we'll have uh I'm sure that to talk about how to further Swiss uh the neural networks uh to microcontrollers tiny little microcontroller combined and five dollars uh together with the tiny engine the tiny influence engine uh that can actually deploy uh the model of microcontrollers. So that will be the schedule for the first part of this course to give you just an overview so let's start with some motivation okay. So let's first introduce this and now perf backtrack which is the Olympic game for AI Computing. So many companies participate in this competition every year uh so that late is the season of about perf just completely concluded last week uh Nvidia Google um AMD Intel a lot of the companies uh participates in this competition.

measures how fast your hardware the algorithm can run a particular workload and a particular given accuracy. Okay, so there are two tracks, two divisions. One is the closed division, the other is the open division. So the closed division basically says you cannot change the neural network, you cannot change the precision, you cannot change the number of weights, you have to run it exactly as it is, just use hardware innovation. Common examples per second you can achieve. There's also the open division, so we can guess another name. You can openly try different ideas, different innovations on changing the model. For example, you can prune the model, you can compress the model, you can reduce the precision, you can do all kinds of tricks as long as the accuracy fits, I mean is well maintained, then it's considered a valid submission. So we can see in the closed division on the same hardware platform, we can achieve 129 samples per second, while in the open division, if we are given the freedom to further optimize the model, we can achieve more than 4000.

samples per second. That's 4.5 x faster. Okay, so the same hardware will immediately reduce the cost. You nearly make it faster. So we are going to learn these techniques. Okay, so how did El media achieve that? Okay, there are three major Peters, which is quantization, that we're extremely pruning, and also distillation. Okay, if you apply this quantization alone, maintaining 99% of the accuracy, the model size is about 600 megabytes, and the speed-up is one combined with protein, which we are going to learn in this lecture and also next Tuesday. It becomes 2.6 times faster. Okay, the model size, it also reduces a lot. Finally, by distillation, but we can use these two laws to recover the accuracy from proving by improving distillation. QAT, we can make it 4.4 and a half times faster. Okay, which is pretty impressive. So that motivates why we want to learn a lot about this, about improving. So today we are going to first introduce the concept of neural natural pruning.

what it is and how how much we can prove almost 10 times smaller um make it a more make it faster more efficient or mobile different steps of how do we do pruning okay what are the granularities we do pruning what is the criteria we can prove uh the neural networks so some basic um motivation is that the memory is expensive we want to introduce the memory uh access as much as possible so a 32-bit ad in 45 another meter takes about 0.1 fifo joules which is pretty cheap uh even though working Point ad is 0.9 access and register file one more application are slightly more expensive as friends access cash access slightly more expensive but as soon as you go to drap okay let's go to the dram it becomes two orders of magnitude more energy consuming okay so it is memory reference that is draining at the battery of our phones so therefore we want to reduce those those memories including both the weight and also the activation okay if you can reduce the amount of weight of a neural

network. You can save the DRAM access to save the power. Therefore, your battery can probably last longer. Okay, so what is truly pruning specific insurance? A density of network into a sparse neural network can remove those redundant synapses, and also you can remove those redundant neurons. Okay, it becomes less redundant. So this study shows the number of synapses in our human brain. Okay, so a newborn child has about 2500 synapses per neuron. Well, he or she grows about four to four years old. This number quickly surges to 15,000 synapses per neuron. Interestingly, this number didn't keep increasing as a child grows. Rhino lessons actually. Adolescence is a period when we go to school and learn most of the knowledge in our life, but interestingly, the number of synapses actually begins to decrease during this period. Okay, so those pruning processes happen. You first have a lot of options for neuron and then a couple less, and humans stabilize during for adults stabilizes to about seven.

thousand synapses per neuron. Okay, so this two new processes naturally happened in the human brain, and similarly, we can do the, uh, I carried out this experiment by in 2015 for modern neural network for Alex snack. So basically, we remove those small connections in very simple criteria. If it is smaller, then we remove it. Remove it means setting it to zero. Okay, since zero multiplied by anything is two zero, we don't need to compute on it, we don't need to store it. Then it saves a memory and saves the computation. Any questions? I should say, but going back to the biological example, I really only shook really a snap, so it's not promoting to revolve any neurons. So do you see in practice for the network that removing neurons is stronger or anything synaps this is strawberry. Yeah, we will cover our both protein-protein in the pruning the weights and also proven the activations and also tell the relationship where if you prove a neuron, you basically pull away all the weights associated with that neuron, so they are not independent.

Hey, so uh, let me uh, start with the experiment I did in 2015, starting with AlexNet that has 16 million uh, parameters. So here uh, this is the original uh, weight distribution um, we explain the model, make it converge, and this is the original based on accuracy as we remove those small connections, those small weights, and we can see the accuracy begins to decrease. Okay, so especially if we observe the wave distribution, those small ones are removed. Okay, the more you remove the mass, the accuracy when you move 80% of the parameters, how much accuracy you lose? Four and a half percent accuracy is lost. So people design I like that or different neural networks just to improve the accuracy by like one or two percent. Suddenly, you lose eight or four percent. That is not good. Can we do better? So we can find you in the model given this model exactly the same, just passing ratio. These connections are bad, these are still alive. Okay, so we want to fine-tune the remaining weights to recover accuracy. Actually, we can do a very good job.

um and do like 75 percent, uh, almost no degradation of accuracy, sometimes even increased accuracy a little bit due to less overfeeding. Okay, so what is the distribution like? In This Moment becomes flatter, right, since, uh, the weights, they have the freedom, they have the degree of freedom, they can adjust, lose weight, some of them, okay, some of them move to the right so that the accuracy, uh, gets recovered compared with just do it once, okay? Training, prune it, retrain it, we can do this process, uh, interactively, prune it a little bit, and then, uh, and then retrain it, and then we can further push the front here to the right-hand side, to the top right corner, okay? About 90% of the weights are currently, well, the accuracy is very well maintained. This is pretty amazing, where, uh, almost no, uh, with no loss of accuracy, you can say, make her model, uh, 10 times less number of parameters. Actually, you have to train it in the beginning, at least for a while. There has been, uh, study to see whether you can

start putting exactly at time zero. That's very challenging. You have to train a couple of iPods. Is this training accuracy or excessive? This is a test accuracy. Yeah, if you try to add me, just like getting into German straight, saying like 90 of these need to be zero, and then you don't need to do an iterative one, because it could be the case that iterative approach gets like a local minimum into it one step. They'll say, Okay, with 90 Proof, this is what we'll do, right? I tried at theoretics. I gave you a very high pronun ratio, removed lots of connections, but the rest of the parameters is hard to recover the accuracy. Instead, if you go a little step like from 15 to 40, you cover them 40 to 30, that works better than directly jumping from 15 to 30 percent. I have a question. You say is the time consumer, is the majority was in the amazing view Ram, so is that possible that we didn't put in the nodes or the acquisition, we just combined with the

same like the like for a model. There will be, say, 100 nodes and so 100 ways, and we combine them two by two within 15 times within the device. Is that possible? Oh, it is possible. That is really the sharing two ways to share the same value. We'll talk about in the next Thursday selection, but not necessarily to share the way but combine away in a safe story place. How do you combine two ways in the same storage space? Uh, like, uh, we don't, we don't need all the ability to this, so I wish I have to wait exactly. Yeah, so you don't have to explain it, but this one definitely. Um, I will talk about how to represent them later, so you need to represent the location of those weights with the iterative reading. Is there like some place where like it doesn't matter like how many iterations you're doing? You're just going to converge to like a peak, like their need ratio of photography loss kind of thing, or is it just like we can iterate as much as we.

wanted like maybe doesn't speak anywhere better right um we find bitterness. Yeah, as soon as you pass a threshold, uh, it's very hard. Even you do some a couple of more information, very hard to um further point by that are familiaries like the the smartly official foreign Auto Norm criteria in this experiment. I needed uh within that layer, you know, everyone prefetched a lot of questions. I'm going to cover a little partner next lecture. That's good you're thinking hello ahead of Elections. Okay, I see um but there are also a newer neural networks like vgt coconut resident 50 and I also School snap. This is what I'm uh collaborating with UC projectally folks back in 2016. You can tell from the name it's pretty small like only one megabyte um so even for this small model that is very small to begin with. We can still compress it right but you can see the larger the models to begin with like major J is 16. The larger the reduction ratio you can have more rules of

compressing or Alexa, which is actually very redundant. You can compress it by open next nine times or like 350 or squeeze net. There's less redundancy. The model is more efficiently designed, but still, there's room of about 3x room, and it will further reduce it. So let's see not only image but also image and language. We can prove this image capture model. Okay, it's given this image. The original Baseline says a basketball player in a white uniform is playing with a ball. And if we prove 90 of this model, it's going to say a basketball player in the right uniform is playing with a basketball. It's pretty much the same. Or this one, a man is riding a surfboard and a wave versus screw 90. A Magnus sweat suite is running away by the beach. Or if we have room too aggressively like this one, the Baseline says a soccer player in that is running in the field. But this controlled by 95, that's super aggressive for Innovation. It says man in the adventure in black and white.

actually refuses no longer correct. Okay, almost getting drunk, so there's a threshold we have to figure it out. So, um, those experiments are also admitted in 2015-2016, uh, becomes super popular in recent years. Lots of companies, um, both industry and academia, are using this technique to accelerate neural networks. Even, uh, recently on large language models, people are using scarcity to accelerate large language models. Uh, this line of research really started back in the 1980s, where there was a paper called optimal print damage. I'm stuck in this area about the beauty of steep learning is that with more compute, with more data, with modern different scales, but even see the principle, we see the math, but existed for decades and got reborn in the context of modern deep learning. Continuing, so here are some other options in industry. For example, um, no, this is my paperback in 2016, where we can try to design a hardware accelerator for sparse and pruned neural networks without having to decompress it. So, this efficient Difference Engine can take advantage of

the sparsity, the weight to ignore, to skip those zeros since zero multiplied by anything is zero. So we can save the compute for both the sparse weight and also those parts' activation. This is that PG prototype for DNE, and also this is a general-purpose sparse matrix multiplication unit. So in media, adopted this with weight sparsity since the A100 GPU reached Universe-like performance in 1.5 measure the birth speed-up. We are going to dive deeper into this lecture, and also designing is now part of AMD adopted as fast as improving in their Wireless AI tool, which is basically coming from abstract chlorine in my stock Hub called DFI Tech, which is now their widest AI platform, which can use truly to reduce the number of parameters reduced by 5 to 15x with minimal accuracy impact. Okay, so given those big overview, let's start a little one more layer deeper into what is pruning and how do we formulate that right. So we have a loss function, the input, and the weight.

okay, previously the weight is noted as W, but after shooting, we denoted as WP. People pruned, uh, we want to make sure the number of nine zeros is smaller than a Target brunion ratio, a target number of nine zeros. Okay, so for example, you have 60 million parameters, you target 10 protein ratio, is n equal to six median parameters. Okay, uh, so we use that to represent the true model. OKAY, using the same inputs, but the weight loss of them becomes zero. Okay, so that's the absolute WP zeros should be smaller than a threshable. And then let's talk about how do we determine the pruning granularity, what is the granular energy, what have you given this 4x4 with Matrix. We can either approve individual elements or we can prove a group of them, right, like two by four, two by one, and all Force granularity. Okay, uh, we're not going to talk deeper than that, just give you a first, give you an overview, and what is protein criteria, um, which synapses, which neurons to be approved, and also the polluting ratio.

how much do we prove and finally how do we have a recover the accuracy absolutely. So when I start with the pruning granularity, which basically says we can improve a different brain analysis from those unstructured to other structure. You can imagine the structure really may have better hardware efficiency, but it can prove less right. So we are going to learn those trade-offs. Given this example, we have eight by eight, I told the wave matrix. Okay, so we can either do this fine-grade unstructured protein where zero as location is completely random. Okay, those nine zeros can appear on either rows or either columns. You can have the most flexibility to prove your model, but it's hard to accelerate because it's irregular, and hardware and parallel computing doesn't like irregularity. There are other approaches, right? All of these eight rows, I think, have we have football force the entire role to be a zero to throw away the entire row. So one of these eight rows, I prune away three of them. Okay, so we have only five rows.

left. What is good about this approach is that absolutely the way Matrix is too dense. It's two Advanced Matrix. You can still utilize potential Matrix Matrix multiplication libraries provided by the vendor. So just let's do this Matrix Matrix multiplication. But um, it's definitely the less likes about to drop. Maybe this way you want to prune it some of this memory. Also have to pronate okay, so you have less flexibility than this approach. But this one is more uh, is more regular. Therefore, is easier to accelerate. Just has given it a smaller Matrix hey. So um, let's take a deeper look at the convolutional layer okay. In last lecture, we talk about a convolution kernel. The weight of a convolution has four dimensions right. So we have the input Channel Dimension which is cu. We have the Oppo Channel Dimension which is C out. And we have the side, the height and width of this kernel okay. So there's four dimensions to give us more choices compared with previously the fully connected layer like you just have CE and also the C out. Now you have

more uh Dimensions, so let's see if what are the pruning uh granularities choices. So remember, we have four dimensions, right? So this kernel size three by three kernel, that's the first two Dimension. We have the input channel, in this case, we have two input channels. We also have the output Channel, okay? In this case, we have three output channels. The fine-grained um fine-grained pruning is basically you can pick any of this from all any of these Dimensions. Any element could be zero, and we can make it um a little I have a little more a better regularity to some structures or some patterns. It's like patches, okay? You can enforce some patterns rather than this completely random pattern, or you can perform this Vector level pulling that you can prove this entire row needs to be zero or nine zero, or you can do this kernel level pruning. It's the entire kirima F3 kernel is either r0 or R and zero, okay? We can go even more regularly in either Channel if either uh remain or prune, so we have

this broad, very broad perspective of pruning regularity for convolution operations. Wait, what's the advantage of retirement? All right, so very good point. So, amida is basically using this kind of iron-based protein. When we give it some pattern, it's no longer completely irregular. So, look at this Matrix. Do you see some patterns? What is some pattern here or some regularity here? It is or spring. I can't. Yeah. Foreign, right? So, what is good about this? You can actually condense it in a way that each row, rather than eight elements, now you're not exactly four elements. You still dance, but you have to pay some overhead to store the index, the index, okay? So, among these four elements, one, two, three, four, we have four elements. Each line element will have to specify which position am I sitting, right? How many choices? One, two, three, and four. Each nine zero elements can sit in either one of four, for instance. So, how many bits do we need to represent? Two bits, right? Two bits, right? So, we did

two bits, two um index to specify uh which is the location in the 190. For example, here this one is zero, right? What is this? It's three, right? What about this? It's one, and this is two. And so, therefore, uh, we need to explore this type indices as uh some actual overhead, but it's relatively small. And so, um, this is what is utilized, and he has energy to architecture, uh, which gives you a 2X Peak Performance. Okay, hope that answered your previous question, and let's continue um the discussion. We were here before, I think we were here before. This is the whole spectrum, and previously we dove deeper into the pattern-based and what are the pros and cons among different choices. So foreign individual redundant weights, okay? And sometimes you're going to get really high induction about the number of weights. However, like we introduced in the last lecture, the reduction in the number of parameters does not directly translate to a speed-up, right? So speed-up also refers to whether it is easy to be paralyzed and accelerated in Hardware, right? So that's why

We also have other pattern-based pruning. For example, when you have a certain pattern like this, we can use GPUs to maintain this regularity. I actually recognize it, followed by The Matrix. So, this is available in the Empire architecture, and we can observe for different benchmarks here like ResNet 50. The previously advanced accuracy standard 6.1 Struess accuracy is 76.2, very well maintained. ResNet X 79, 79 exception model SSD for detection, Mask R-CNN, FairSeq Transformer, and BERT large across vision and also these language models. These accuracies are pretty well maintained. Foreign, let's go to um The Other Extreme, okay, the channel three. So, for channel pruning, it was probably the most uh regular one, right, being removed by the entire channel. So, after that, you get a very dense matrix uh for convolution. You just have one um one less number in the one of the dimensions, okay. So, this is the easiest to accelerate and uh for example on your phone without any accelerator or even a Raspberry Pi, you do that kind of pruning.

you can immediately get a reduction for the memory and also the compute, but the bad part of this is you can throw legs sometimes 30 percent. Okay, um, so that's the trade-off. This one like nine percent, that's like, that's nine times versus like 30 percent. Okay, so um, let's look into some cases about Channel cleaning, which is the last case. Although you prove uh, you can draw less, so this is very wildly adopted in the industry. If you um, go to like a mobile company, this is probably the technique that is widely used since it's just so easy to accelerate. So this is our original uh, different layers. You have five layers, each layer has a certain amount of channels. You can just prove different channels to make it more um, to make it have less number of channels given the same uh, pruning ratio, the ultimate preneurship, the same ultimate pruning ratio. We can either uniformly reduce, shrink uh, different layers, or we can wisely choose the sparkly ratio of different data. We'll discuss in the next lecture and how do we find the sparsity.

ratio for each layer and just put a conclusion. Uh, here in this lecture, this is the latency versus the accuracy tree loss. So, which point will be ideal latency accuracy? We want low latency and high accuracy. So, ideally, at this point would be the best. So, this kind of wine stage using the premium ratio of a different layer is better than uniformly sampling the different layers. Okay, so we'll introduce this technique actually done by around the um how to select automatically select the protein ratio. Hey, so um now we want to talk about what is the uh pruning criteria and there are so many synapses and there are so many neurons that can be candidate of pruning. Then how do we select them? Okay, so we'll take a break and then after that we'll jump into the section three to talk about the protein criteria. Take a five-minute break returning at four. All right, welcome back. So, let's continue to talk about the section 3, which is the pruning criteria. So, what synapses and which neurons should we? So, let's see this example. Okay, so we

have a simple multi-layer perception (MLP) that has three weights: 0.1, W2. We combine three inputs together and pass it through an activation function. Okay, so imagine the pyramids. We have 10 minus eight, 0.1. So, this simple one layer of perception is basically 10 x 0 minus eight x one plus 0.1 x two. So, by intuition, if only one weight will be removed, which one are you going to remove intuitively? This morning, right? So, actually, that's exactly the starting point about selecting which synapses to pull, basically based on the magnitude. And also, that's exactly what I do in 2015. So, we try to find a very simple way of drawing those connections, like you have four weights that you just calculate the magnitude. Okay, so we just find the magnitude of these two are pretty large with five, one, and two are pretty small. So, we keep those larger ones. So, this seems super simple, but interestingly, that's the method that is

adopted by industry for the last five years, right? So, a lot of fancy techniques. We always want to find a trade-off between the effectiveness versus how easy it is to integrate into a product, right? So, doing some research versus something that is really able to land into product. They have a very solid relationship, right? So, are you ready? Um, yeah. Tell the people that um, how do they find Effectiveness? That's basically uh, the accuracy or the performance divided by the complexity. Okay? If the code is super complex, you need some predictor, you need all kinds of complicated stuff. That's very difficult to make it into products. The other are other methods, right? For example, uh, we talk about different cooling granularities. Okay? So, this example, we prove at the fine-grained program, right? So, here is how the flexibility to Pro individual AIDS. But now, we want to prove a whole row of weight. So, for magnitude-based pruning, how do we generalize that to a core screen free? Very simple characteristic here. We can just sum up the absolute value across different row.

right, so three parts. Two is fine, and here we have six, just summing them up, and we've thrown away the smaller magnitude. Okay, there is another criterion when we are doing a core screen pruning that involves multiple elements here. So here we have three and two. Now we can calculate the auto Norm rather than the R1 Norm. Okay, so previous question, a previous example, we calculated in the R1 Norm. So this is the R1 Norm, just summing up the absolute value. So you can also perform the other two Norm magnitude space, the queen. So we can calculate Auto Norm 2, very simple math, and we can pull them away at this model left and turn out how to normal. Our Norms became the same result. Okay, both of them are pruning the first zero. So in general, there is the lp Norm. Okay, so w i to the power of P, sum them up together, and then to the power of 1 over P. Okay, so um, that is a more general heuristic, other um, more um, better approach for convolution neural networks. So in

convolutional neural networks. We have many filters. Okay, um, foreign approximation. So low-rank approximation is orthogonal to pruning. So you can definitely remove, uh, those, uh, just keep a few ranks. Like recently in Laura, in a large language model fine-tuning, that is a very widely used approach. You can also actually think of it as a special case of pruning, but not necessarily the same because, um, if a weight matrix is low rank, uh, you multiply them together, the dimension is still the same. So usually people talk about like pruning, quantization of um, low rank, and distillations for different techniques. Let me say certain elements. That's why I say that those are orthogonal techniques to control on top of a low-ranked matrix. And also, you can quantize on top of a proved matrix. And I can use this relation to supervise pruning, quantization, different loss losses. All right, so um, actually we can learn a ceiling factor if we are doing a channel uh, channel zoning. Okay, so um, we modify the results of this whole

Channel okay maybe uh like 256 by 256 uh. We can multiply it by a scaling factor okay. So when we are doing the learning we want to encourage this scaling factor to be as close to zero as possible okay um. And then when we are doing improving um we see the absolute value of the scaling factor. If the scaling factor is pretty small then we improve those channels. So here is the example. This is before pruning. We have one two three four five channels and these two are pruned. So we are only left over with those big scaling factors and luckily this scaling factor is already there in the batch normalization. Batch normalization layer which I if you remember we covered batch normalization in the previous lecture okay. So we can just reuse uh the scaling factor in the batch normalization native to determine which channel to. There's also the second order uh based pruning method. So uh if we have people into the mask the change about the loss function would be. This is the original loss function given the input given the original weights and now we are given

the same inputs to try to prove the weight equal to the original weight answer are subtracted with a Delta. Okay, Delta means you are changing from your original weight to zero. Okay, so according to the Taylor expansion, this is the first-order term, this is the second-order term, this is the third-order term. Okay, so um, the last term, okay, so um, the last one is removed, assuming is pretty small, and then assume the Metro Council has converged. So the gradient, the first order, is also removed. So we are only left over with the second order and assume each parameter is independent. We can also remove the closed term. So we are only left over with um, this second-order term. Okay, so uh, remember this, this Matrix, and this is The annotation uh, which we are going to put it again here, right? So we just need to calculate the hashing Matrix. Okay, but actually, Matrix is difficult to compute. So finding the full solution, the exact mathematical exact solution, is quite computationally heavy since the second-order derivative. Um, so as an alternative, uh, people have derived the various approximations.

rather than computing the hashing matrix, so I leave that off to future breaking. Hey, so previously we talked about our weight only, and that's the question in the beginning of this lecture. We also have the opportunity to prune those neurons, okay, both for linear layers and also for convolution layers, okay. If we can find the neurons that are not important, the more accurate we can find those, the more accurate we can, the better performance we can get. So in this example, uh, this is the second neural as pruned, okay, and I mentioned this highly correlated with weight degree because as soon as this neuron is pruned, all the weights associated with this neuron are all from right. So this neural pruning is basically a core screen wait for me. The neuroprone is basically a horizontal acidic right-hand side. Previously, we have five input nodes, neurons, and four output neurons. So what is this image of the weight? Um, so we remove one of the output neurons, which corresponds to all the weights associated with that new rock is removed. So that is in my clip, of course.

with the course when weight protein similarly for convolutional neural network rather than two Dimensions now we have four dimensions right kernel height kernel with child input Channel upper Channel um so we are doing Channel pruning in a convolution layer for example um right so this third channel uh gets prone that's the activation including the neurons and also the sixth channel is also Pro right correspond control first and as a result uh the third and also the sixth Channel layouts thrown away okay so that's also a core screened wait holding therefore neural protein is highly correlated with the way to produce so let's see an example of a convolutional neural network and how do we determine uh which uh activation to group which neuron or which activation um so usually we have this random function what does Randall do uh if it's smaller than zero the activation will be zero right so like in the switch format it is four by four the resolution is 4.4 and here we have a batch of two imagine we have two images right so that's the

batch demand we mentioned, and also we have the channel Dimension. We have three channels; we can imagine like RGB. You have three channels, okay? And imagine we have image one, and this image is two images. And now our task is to determine among these RGB, these three channels, which channel, or which, uh, yeah, which channel should we throw away, right? So one simple heuristic is to examine the number of zeros in a channel, okay? But you have different batches, so we want to look at that holistically across different batches. So we want to first define what is average percentage of zero activation (AP on Z), okay? So, uh, rather than giving a very complicated math description, I always would like to introduce by a simple example. So here, um, how many, um, what is the average percentage of zeros? How many zeros are there? Because we have one, two, three, four, five, okay? And we have one, this is China zero, okay? And also, we have a second image, which is here, right? So we look at also channel one to see for the second image how many zeros are there in time one, one, two, three, four, five, six. We

have six zeros in China. One okay, so we have five plus six. All together we have 11 zeros for channel one across two images. Um, I mean in order to calculate the ratio we need to divided by the total number of elements. So here we have four by four we have 16. Here's another form by four. So all together we have two times four and four. That's 32 elements. Okay, so lastly uh ratio for the average percentage of of zeros in the first channel. Okay, and similarly we can do the same calculation for the safety Channel and also for the third time. Okay, for the second channel uh one two three four five and five times zero. So five zeros and here we have one two three and four five six seven. We have seven and zeros in the second image. Okay, so we can calculate the average percentage of zeros on the second Channel which is 12 over 32. And similarly we can calculate that for the third of the channel here and here which is 14 divided by 32. So given these three numbers of

percentage of its zero. Which channel should we prove there? Um, since 102 has a lot of zeros, so you zero them out. You have minimal change, right? So we want to improve away. Remember the channels which has mostly zero, right? It has the largest ratio of zero, meaning that if you set all of them to zero, it will have minimum impact for the final result. So this is how do we do activation food. Okay, so by this time, we covered both the weight running and also the activation pruning. There's another way called regression-based three, okay, which tries to minimize the Reconstruction error of the current corresponding layers outputs. So previously, we were talking about the loss, which is feeding input. You get the output, passes through the entire neural network, you get a final loss. So here, we are trying to examine the individual one individual layer, okay? The input, we have the wave, you have output. Now, this is super helpful actually, um, with the recent wave of large language model, you know what? You know why? Because they are so computationally expensive, like one standing five billion.

parameters is super hard um to fine-tune to run through the whole network to reconstruct the weights by back propagating into each layer. So people deny that in a smarter way that I can do it locally. It just looks at one layer, just looks at one layer. You try to minimize the change before and after, okay? So before um you have a 2X 2x4 inputs by size of two and input Channel. You have four input channels and now you have a weight Matrix that projects four input channels into eight output channels. And as a result, we have two by eight um output tensor. So whenever you learn a new neural natural architecture, it is super good to understand the channel, the bigger the channel Dimension. So that's why rather than using math, I want to put in individualization to give you an idea. So now we want to approve one channel. So rather than four channels, we want to reduce one channel um why not from this way because this is the patch Dimension first input second input in later part of this lecture. We can also talk about

token Cooling like in natural language um word is roughly I think about it as a token right so not not all the words are equal some of them you can neglect it but you can still understand the meaning of the sentence so there's also opportunity to approve in this Dimension which will cover in a later part of this lecture [Music] the second channel in the input means for the weight um the second Channel also the um this Dimension corresponds to this dimension in Matrix and multiplication and how do we determine which channel uh control so we're formulate that as out for product okay so um outer product means we do this and actually times this Vector what is the output dimension two by one one might oh my two by one and one by eight you get two by eight so that expands to an intermediate output which is of Dimension two by eight similarly um the first the second Channel also can multiply with the second channel of the weeks and get a two by eight uh intermediate outputs okay how many do we have to move out together we have to do four times

first of all, the second one on the fourth line. Okay, so that's why we are solving up different intermediate activations together. Okay, from zero zero to uh c i minus y. Okay, and we get different partial matrices. So the trick we do here is we wanna for each partial Matrix partial Auto multiplication. We want to have a coefficient that is learnable from the beta and beta. If it is zero, it means we get rid of the channel multiplied by this title, right? Um, so we want to make sure the number of non-zero channels is smaller than a given threshold. Okay, um, so what we do here is that we can iteratively uh first fix the W and then learn the beta. Okay, and then we'll fix the data to uh to learn the W. Learn the w means we can adjust the weight to make a fit after the pruning. Okay, so by doing this process is relatively we can figure out which way. So last year, we want to talk about this large language model. Was not that popular, but given the context of large language model, this kind of

technique would be quite interesting since you don't have to back propagate all the way to the end, which requires like a super big GPU, but here you can just solve, um, reconstruct the weights for individual layers. But you can imagine this has drawbacks, right? You may not have a full holistic picture about the entire neural network; you're just reconstructing one individual layer. So always there's no frame latch, that's the beauty of efficiency in different computing. I don't sure, so isn't that really similar to the, uh, there is eventually the PCA investigation area, like you know, similar to the elsewhere, but ultimately important, uh, the Reconstruction we mean really assign tuning the weights, so it's not exactly the PCA stuff. I mean, so SPCA is also minimized a lot of the error between the original Dimension. Oh right, right, using this iterative approach, right? Yes, right, right. They are both are done yesterday. So before concluding the lecture, I want to actually, uh, show a pruning demo, uh, which I believe will be a lot more fun to see in practice. How do we Pro on your

network. So, let me pull the screen to the left. All right, so let me reconnect. So, here we prepare. Um, I miss the data set. Okay, so for you, for us to play with uh, pruning and improve the net. So, let's first visualize the demo images. So, basically, we are doing uh, digit classification like zero. This is the classification result and three. This is the classification result. And now, in this example, uh, we are just visualizing the data set. It's just 100 written digits. And now, we retrieve a neural network. At least um, maybe you have to set up, run it from French. By the way, this is the Google collab where the GPU is just in the cloud. Everyone can access. We are going to use Google collab to do the homework. So, you could haven't used it before. Um, highly recommend. You can try Google Google collab. For example, try lab zero to make sure you are familiar with that. And also, Google collab has two versions. One is the free version, one is the paid version. I think the free version is

enough for us to complete all the way from Lab One to level three. Visualize an image. Now I train the model using the free version of Google Colab. I think you're going to relatively a lower end of the GPU, but I think still the speed is still reasonable. You can still pay ten dollars a week, a month, to purchase the Google Colab Pro. You can get a faster GPU. I think that's necessary when we are doing level four since we are playing with large language models, and I think it's worth to use a higher education. But it's doing the work. It was three for one iPod, and I think we're going to train like four or five eyebox. Then it is going to converge for almost 99. I think it's pretty much coverage. Just give it one more iPod to make sure it converts right. It's pretty much converge. So now we can evaluate the accuracy of the original model B4 probably. So we get it 19.97. We can assume it's 99.

uh accuracy given the digit. This is the prediction; they are all correct. So, um, let's try to prove the model. So, I give it a sparsity ratio of uh 70, and let's see how that works. So, using the pruning criteria, I remove um a lot of parameters. We remove actually like 70 percent of the parameters. I only keep 30 of the plant sure, and as we can see, the accuracy slightly dropped, right? It's like a job to 96.22. How about we increase a premium ratio to 80 percent, meaning that we are removing 80 of the prime interest with only 20 percent off of non-zero, and you can see the accuracy immediately dropped to only 76 percent. And for example here, given this image, it should be eight, right? But approval model size is five, so let's see the power of financially. So, I've removed 80 of the parameters, and then I can do some fine-tuning of two eyebox overplay to recover the accuracy. So, the fine-tuning implants started uh. This is the first iPhone.

this uh like more trading right. The only thing we did uh is you mask away those parameters that's thrown away and fine-tune those remaining parameters. Other other aspects are exactly the same so which training just magnitude of the basis no individual running away in individual ways simple method okay so um as far as accuracy is already 99.89. So let's evaluate the accuracy again on the test set. It's 98.89 right pretty much recovered accuracy and here the digit 8 is now correctly classified as eight okay so showing that although we are removing 80 percent of the parameters is too powerful enough where the remaining 28 is a fully recovered in accuracy. Oh this is the Google Cloud doesn't have like the end here whatever DQ architecture like you would expect more speed level if you actually reflect a more modern spot right. This one you have to pay it to use an Empire GPU but in order to do that you have to have the user tool it will make it a two to four spicy all right. This is the regular uh the the um

irregular um random sparsity. Yeah, so in order to achieve actual measure the speed up, we need to uh. Our next lecture, we'll talk about those techniques that can accelerate on directly and responsible Network. We can go more aggressive. We want to break break it, so let's give it a higher threshold, maybe 90. Okay, 10 percent of those that's find the real life where we use the Dropbox instead of using the an example way to find like you know understand where you use the actual Parts if you drop some levels are different as different motivation. So job hot is designed to prevent overfishing or training, and pruning is designed for reducing the computation for uterus. So we can use an example with example. We need to replace what do you mean by that like we're running different ways like um Matrix that includes past Matrix to get a tensor Matrix that's different. Just open that question thinking about how I think being when you're pruning, you're effectively exploring all available parameters and then foreign also different iterations are dropping.

out different stuff, so and rely on the fully Advanced model in the end to do the prediction. So job call is for training, but prune is for inference. So let's see after we remove 90 percent, um, the accuracy is only 20 percent, and that's kind of fine-tuning for two ad hocs and see how much it's been recover. And by the way, this will be the starting code you can reference using when you are doing the homework one, which is basically going to implement this pruning process, this fine-tuning process, which will be pretty fun. Functions I've tried this thing, we didn't get a, it was still just masking away those waves, so you are doing all the computing. We'll talk about how to get measure speed up. We have to write special kernels. Okay, so it's for our box of fine-tuning. That's not has the accuracy evaluated on test side is not eight point five percent, which is actually pretty good. Um, so we can do it more aggressively until let's make it uh 95 maybe 99. That's

right okay this is wrong. This is wrong. That's as you expected. Yeah, it's predicting everything at school. I think it's actually better that it's random because at least everything to you gets a 10 accuracy. An accuracy should be very close to 10 percent, right? The accuracy is like 10 points, like 32, exactly as expected. So, let's test the accuracy. Apps are fine to me, so it's only a 25 percent increase, but cannot fully recover, right? So, probably that's the limit, I think so. Hopefully, you can use this code base to help you to finish homework one, and also you can use the code base for the final project. For example, you want to explore other aspects of trade, right? How does the approved model generalize? Is it to include the model or easier or harder to get a zero attack? For example, those are the other aspects you can play with by doing surgery on the neural networks. Okay, all right. With that, I think that's all for today's lecture. Thank you. See you next.

EfficientML.ai Lecture 3 - Pruning and Sparsity (Part I) (MIT 6.5940, Fall 2023, Zoom recording)

MIT HAN Lab: Hardware, AI and Neural-nets

Accelerate Deep Learning Computing

Group website: hanlab.mit.edu
TinyML Project: tinyml.mit.edu
TinyML Course: efficientml.ai, streamed via live.efficientml.ai every Tuesday/Thursday 3:35-5:00pm ET, fall 2022.
