So, when other people move their lower bodies, does it look a bit strange? Sometimes the left three might jump up and down, and sometimes it doesn't move exactly as intended. However, by frequently sampling and learning the input and movements together, and then matching them to the codebook, it definitely looks more accurate. I once tried to estimate how well it follows the road, but when compared with other methods, the error is quite significant. However, it's quite accurate in doing all of that. [MUSIC] Hello everyone, this is Engineering Error for a month. Horizon Worlds was introduced in the metaverse world by Meta, but headsets haven't spread widely yet. Nevertheless, in Meta's Reality Lab, they continue steadily with technologies related to the metaverse.

I am researching, and this is the paper that will be presented at SIGGRAPH 20248. There is one significant thing that I brought with me. It's a technology that can accurately estimate movements even in the legs just by wearing a headset and holding controllers. Actually, at first, we removed the leg part and it appeared in Horizon, but later the legs were added, and there was a lack of accuracy, so when I moved in a certain way, the character moved differently, and this was a problem. However, we found a way to do this very accurately. The author uploaded this video while submitting the SIGGRAPH paper. When we play games like Beat Saber or move that character, we actually have control and hold a gun, but sometimes the leg part feels awkward. But depending on my pose,

I found a way to estimate it well so that when I move in a certain way, it will estimate how the legs will move like that. In the past, there were studies like that, but this time it clearly showed improved results. The point is that it generates motion with very high accuracy through just a few sensors. To understand why this is amazing, you need to look at how it used to be. It is commonly called three-point control, where you have a headset, and two hands to control three points. When you had these three points, the issue was how the legs would move. In the past, you had to separately learn the mapping between motion and control signals.

Motion prior is like saying, If we move like this in advance, our legs will move like this. There must be data that says when I make this posture, how should my legs move, or when I make this posture, what should I do. We need to map each movement precisely when we actually control it, based on the motion data prepared in advance. So, we have to separately train these two things. First of all, it takes a lot of time to learn each one separately, which is not efficient. Moreover, the memory of the headset is very limited, so if we want to use it in reality, we need to reduce latency and respond quickly. To do this, we need to reduce the amount of memory and respond quickly, which often leads to unrealistic situations. It feels a bit awkward.

Yes, there are limitations. What's moving right now on the far right is the restricted one. When it moves now, sometimes it floats or makes strange movements a little bit. At that point, it can become a memory latency issue or various other things. Of course, it has been continuously evolving, so there were limitations in accurately estimating this posture in point tracking issues. So, if we look at some papers around here, it's written like this. Categorical codebook matching for independent character controllers is shown here. Instead of going into details, I will explain the significant ones among these drawings. Therefore, the existing method ultimately required separate learning of the user's movements, that is, control, and the actual movement of the avatar. Now, these are integrated for joint learning.

Saying I will do it is a basic difference. When we learn to imitate something, we learn hands separately and feet separately. It's not easy to combine them later. But if we start together like this and learn complex movements, it works well. If you interpret it like this, it's good. Here, you can see what is called a codebook. Ultimately, you take samples like this and use the data that the user actually moved. The input is considered as data where the head and hands moved, and the corresponding output is directly related to how the lower body moves, or how the virtual character in the virtual apartment will move. When I moved like this, if the output is that the virtual character moved, it's called codebook matching. When you look at the part called codebook matching, what exactly are you matching? When you see the movement above, is it matching with the actual movement or the character's movement?

There will be some kind of motion, right? If there is movement, these patterns that they move are similar. In fact, they adjust quite similarly. Specifically, it matches the probability distribution. There is a probability distribution of possibilities that can occur. There are probabilities of doing this or that, various data movements. They roughly adjust to shapes like that. So, further on, sampling each motion in parts like this, can you see that? So, if I say that three points are like this, like this, like this, because of these three points, then, following the root trajectory, how the legs will move, it is created for each lower body part through autoregression. This may seem obvious, but if you consider adjusting each sampled one like this.

I think it will work, so when others do the lower body, does it look a bit strange? On the left, there are times when it jumps up and down, and sometimes it doesn't move exactly right. However, by frequently sampling and learning the input and movement together, matching them in the codebook, it definitely looks more accurate. I tried to estimate how well this one follows the path, but when compared with other methods, the error is quite significant. However, it is quite accurate in doing all that even with just the head and controller. By continuously sampling and seeing that it accurately follows those positions, you can see that it is doing so. Compared to the red, the blue is almost matching well.

This will eventually be more accurately expressed when more sensors are added, right? As the data increases, there will be much more to learn. So, based on the history of past movements, how to control the future will be predicted. It is estimated how the body will move, taking into account how it moved in the past. As you can see now, in games or in graphics VR, when obstacles appear, you have to avoid them, right? You have to dodge them. So, what is called avoiding a collision ultimately means that a virtual character behaves in a way that does not collide with other objects. For example, like a wall, you have to avoid overlapping, so that when we move forward, we don't just run into a wall or something like that.

If you do that, it's weird. You should do those things well because now the sense of humiliation gets better. In order to implement that well, you need to divide the surrounding spaces of the character finely and divide them into grids, predict collisions in advance, and avoid collisions, so that you can perform actions to avoid them. So, when the character touches something like a chair or touches a rabbit with a finger, they can move smoothly without colliding. It is said that they are adjusting it so that it can move naturally without colliding. It shouldn't be like the hand goes through. Therefore, in order to avoid that, it will be done by allowing it to move. Nowadays, rather than other papers, interesting and the latest technologies are being released in big tech companies like Siggraph, so I looked into it. It may seem difficult to estimate legs, but I think it can be done once you learn.

Furthermore, when we successfully train the entire dataset through joint learning and achieve accuracy, we can say that it is now possible. Even with a limited number of controllers and a small amount of sensor information, we can create something very realistic. Therefore, we can expect various games and applications derived from it. That's all for today's summary.

메타가 메타버스 했다...! 실시간 VR 캐릭터의 하체까지도 자연스럽게 구현해내는 신기술 공개

공학박사 '에러'가 전하는
최신 공학/테크/IT/신기술

문의 : Unrealtech2021@gmail.com

메타가 메타버스 했다...! 실시간 VR 캐릭터의 하체까지도 자연스럽게 구현해내는 신기술 공개

Scripts