Hello world. It's Suraj, and just a few days ago, Facebook's AI research team released a demo called DensePose. They were able to map out every single pixel of a human body in a given video, and not just for a single human, but for many, many humans all at once. Even more impressive, they were able to do this in videos with lots of distractions and general chaos on a single GPU. Them, nope, just deep learning. This is pretty cool and has tons of applications, and we're gonna learn how they did it in this video. Creating 3D animated characters is a lot of work. The tools to do so are expensive and require teams of people working for months on end. Big production studios with the available budgets are able to track humans and convert them to animations, but for people without big budgets, this technology will democratize that ability. Remember my video on the deep fakes algorithm? Well, this can be used for a full-body version of that. If we can track a human, we can swap out their entire body with a different one, leading to some surreal scenarios like Geoff Hinton.

charging into Mordor, instead of Aragorn. Augmented reality applications could use this to label people that you see in real time, with all sorts of statistics. And it would give away for machines to be able to better read our body language, for everything from sentiment analysis to full-body tracking, to virtual reality gaming. This is the new state of the art in human pose estimation, and to understand how it works, we're gonna have to try and work through the thought process the authors had, both in terms of building on previous discoveries and trying different methods. So let's say we have a video of a human dancing. This is a 2-dimensional grid of pixels, but when you or I look at it, we're able to tell that there are indeed 3D objects that are being represented by this 2D grid. We want our computer to have this ability as well, and in a way that can visualize it, meaning we'll want to transform this 2D human into a 3D model. Once we have that 3D model, we can do whatever we want with it, put it into any sort of scene.

Swap it with another 3D model, change its features. Lots of possibilities, but all of these possibilities depend on us being able to construct a 3D model of this human in real time that updates its movements as the human moves. We want to construct a correspondence. This is a computer vision term that is a measure of how well pixels in one image correspond to pixels in another image. In our case, it would be a 2D to 3D image, and we don't want any holes in our image. We want all the pixels to be as close together as possible, so we'll call it a dense correspondence. And we'll just focus on doing this for humans for now. To create a dense correspondence, we'll need to perform some object detection, object segmentation, and pose estimation. That's a multi-level problem, so we want to start with a method that's as simple as possible. We've got so many different avenues to go about this, but ideally we could use some sort of labeled data set because having a label just makes training

machine learning algorithms much easier since it just involves a learning a function that Represents the mapping between the input data and the labels. In this case, What would the ideal label datasets look like? There are tons of labeled image datasets out there these days. Pascal, CIFAR, ImageNet, these are basically big collections of images of a variety of random objects, each with their own Particular label. These datasets have been a catalyst for all the recent progress in computer vision algorithms. But we don't want to just label a human if we see one in a video. We want to be able to create a 3D model of one. There's not really a dataset of human images with the label being the 3D model. So we'd have to create one ourselves. And it would involve humans manually creating annotations that relates 3D images to surface-based representations of the human body, and that's exactly What did they basically asked the annotators to annotate regions? Corresponding to defined body parts like the head and the legs, all of these annotations were labeled with their corresponding

3D body parts, which acted as the label. They did this for 50k humans, which summed up to be a total of five million manually annotated correspondences. Once we have our data set, we're gonna have to decide on what model to build. We know that deep learning is the state of the art when it comes to classification and that convolutional networks are the state of the art in image classification, but it's not just classification we're trying to do, but regression as well. What's the next move the human is going to make? The next question would be, has anyone done something similar with a convolutional network? Of course, luckily there was a recent architecture called dense reg that did this for all sorts of objects and got decent results, so we could start with that architecture. In the first step, the network will classify a pixel as belonging to either the background or one of several region parts that gives a rough estimate of surface coordinates. This is essentially a labeling task that can be trained using gradient descent. In the second step, a regression model

will indicate the exact coordinate of the pixel within the region part. A more formal way of saying this is that in the first stage, it will assign position L to the body Part C that has the highest likelihood as calculated by the classification Branch, and in the second stage, it will use the regressor to place the point L in the continuous coordinate pair UV that is a Parameterization of Part C. C can take 25 values; one would be the background, meaning that P is a 25-way classification unit, and we can train 24 regression functions, each of which provides 2D coordinates within its respective parts. Both the classification and regression tasks are trained by minimizing a respective loss function, but the regression loss is only taken into account for a part if the pixel is within a specific part. So, our network can work, but it requires a lot of tasks for a single network, like part segmentation and pixel localization. They use a technique called Region of Interest pooling to create different regions and feed the results, seeing features into region-specific branches. This decomposed the complexity of the task.

Into controllable modules, all of which could be trained jointly in an end-to-end approach. So it's a fully convolutional network on top of region of interest pooling that is entirely devoted to two tasks: generating a classification and regression head that provide part assignment and part coordinate predictions. To improve the model, they used a technique called cascading, meaning using a collection of models using all the information collected from the output of one model as additional information for the next classifier in the cascade. The output of the region of interest a line module feeds into the dense network as well as networks for the masking and key points tasks. The first stage predictions from all tasks are then combined and fed into a second stage refinement unit of each branch. When experimenting on novel videos, their architecture worked well across the board for not just one human, but multiple humans. Unfortunately, they haven't released the initial source code just yet. But we can recreate what they did using the details in the paper. Also, the dataset they created is open source, so definitely check.

That out three things to remember from this video. Dense pose is a new deep learning model that can estimate 3d models of multiple humans from just a video on a single GPU. They use a collection of Convolutional networks to do this, performing classification for assignment and regression for coordinate predictions, and They open source the data set they created mapping humans to their corresponding 3d models. Please subscribe for more programming video and for now I've got to make a goodbye pose so thanks for watching.

DensePose - 3D Machine Vision

Hello World, it's Siraj!  I'm a technologist on a mission to spread data literacy. Artificial Intelligence, Mathematics, Science, Technology, I simplify these topics to help you understand how they work. Using this knowledge you can build wealth and live a happier, more meaningful life. I live to serve this community. We are the fastest growing AI community in the world! 

Join my AI Sports Betting bot community: https://www.wagergpt.co 

Twitter: http://www.twitter.com/sirajraval
Instagram: https://instagram.com/sirajraval/
Facebook: https://www.facebook.com/sirajology/

If you found my videos useful, join my community: https://www.chatgptschool.io

Some Research -https://drive.google.com/file/d/0BwUv84lNDk72Q1gzaXgwR2U3U2NWVlZSOFk4amZIRmV1QXI0/view?usp=sharing&resourcekey=0-WgUkgIPZd-qCklnTUElNlg 

In the event of my demise, you must finish what I've started here.

DensePose - 3D Machine Vision

DensePose - 3D Machine Vision

Scripts