okay, it's a good pleasure to be here. Thanks for all the wonderful workshop. I'm going to talk about connecting six post with double manipulation. So, uh, let's start with mod6 post estimation. As we work on this problem here, give an input image. Suppose the image is taken from a camera, and here model base means that we have the 3D model of these objects, and the setup is that we have a coordinate frame from the objects. Now, the goal is to estimate the location translation between the object coordinate frame to the camera frame. So, translation means that where the objects in the camera frame, and orientation mean that what is the rotation from the object coordinates to the camera coordinates. So, what can we get from the six process mission? We can know two types of information. The first is we know the object position in the camera, that is the 3. We also know what's the orientation in the camera. So, we can realize this information by using input image. Suppose we can estimate the object 60 post. We can have the P Cloud RGBD camera. Then we

draw this XYZ axis in terms of the RGB color in the access right. We know where the position orientation of these objects. If you have the 3D model of this object, you have to render this model and overlay it with the input image. You can project the 3D model to the input image. So, F six post estimation, we can get this information about objects. Now, let's see what's the applications. First of all, we can use T of the PO estimation for double manipulation. For example, in this case from the lab previously, we have us to pick up objects y objects using RGB camera. We can also do augmented reality right in this case supp estimate the post of the CH trans the power on the objects or can use SE Pro estimation to do some detection of B the case of the industry objects. But to this Focus, I'm going to focus more on global manipulation. So, in the TRU P Mel based manipulation, we have this perception planning and a control pipeline in

perception. We can focus on the six PO estimation problem. As before, we can detect objects, estimate their pose in 3D, but in planning, we can tackle this class planning and the motion planning problem. For example, I can set up the planning thing using the 3D model object, then plan the trajectory of the robot to pick up objects. So finally, in the control stage, the robot just needs to follow the planned trajectory and grasp objects. In this case, the arm will pick up the picture based on the planning. So let's start with the perception problem, the 60 PO in the traditional way of doing 60. There are two types of methods. The first one is a feature-based approach. This approach basically builds a 3D model of texture features. Then you can get a 2D-3D correspondence from that. You can solve PMP to get the 60 pose or template. But model-based methods, if you have a 3D model, transfer image of the object model, you can compare this.

template to the input images. Uh, so it's interesting to know that all the Dearing basic approaches still cation from this traditional 60 post estimation method. So next, I will Qui to talk about some previous work I have done for six po. Uh, in AUST 2018, we introduced the first end-to-end 60 poiss Network called poan Pro input image and then outut 6 multiple objects in the input image. My hand objects, symc objects, and the clu different objects. So, so this is the first an post Network get input image. We can use a feat except to get some IM features. The first brch of the network trying to segment objects. We get the sematic S mask for the second Branch. We ask each pixel to predict where the center of the objects. So, there are Center Direction. Each pixel predict the direction to the center. And we also predict the dep of the objects. That's the D of the translation. By using this prediction with same direction centers, we can do a half voting. Uh, in the network. So a half

booting layer in the network so it can detect objects and estimate object center that's half for occlusion. So lastly, we can detect the network. We can just have a sub-network to predict the rotation of the objects, that's the rotation, 3D rotation, different parts. But by combining all the branches, you can get the 60p of objects. So here, some viation try Po in the Dr work. You can detect these W objects. You can see the segmentation, the Ping box, the 60 Post in terms of XYZ coordinates, and the overlay rendering. You can see the object is still uh Jing because this is a one-stage prediction network. The regression of the post is not that great. So later on, we introduce the D, that's an intuitive requirement network, that's able to get more ACC post. But the idea of is G and compare as most of previous work has talked about theut image. We have the initial post estimation that can come from any post estimation method. We have the 3 model where we can G the 3D model with get a

image then work which takes the G image and an image and gets a data post with a data post can add it to the original post to improve it and this performs the E requirements for post one and vend again input to a network and another post then we Define the post right so this thetive process can inter improve the post and code report from 2020 the best method is also using this F requirement method the in term of numbers and YB video can see that P can f person L Maring especially in the rgbd case it can be IP requirements so in the world you can see that the DM get a very stable post estimation that's due to the inter requirement the network however as is approach it's estimate a single post estimation there's no uncertain or distribution post soat we introduce the postp it's a z black the future for six WE Post tracking the idea is to apply a fter to the 60 post demand so can get a

temple estimation for Post tracking, as well as certain estimation, but in po RPF we have a particle, and then the particle consists of two parts: the 3D translation and the orientation distribution. Given the 3D translation, we can localize the objects in the image. We can get the object ROI, but to estimate the orientation distribution, we can compare the ROI and use it into an incorporated computer feature for the ROI. Then we can have a look by gendering the 3D model into a large number of template images, so we can compare all features to the PO and get a distribution of each orientation of the input. And this is simulated by Mar work on the Ling implicit implication of the rations. So, the benefit of postp is we can do Post tracking. So, in this video, you can see the camera is moving, uh, so can track all the 60 POS of all these objects. But the video here is with SDF-based depth requirements. Okay, so lastly, I'd like to mention the GD net, that's the over.

the best me in B CH last year. So, this is a direct Network for post estimation that's got a very great accuracy for the GDR. And that it's take input image, then object detector Euro and EUR object detection gets the object ROI. Then the middle stage of the network predicts for each pixel what the 3 coordinates of object, what a state on the S model of that pixel, as well as the mask. Then by using this me of the network, we can get, they can get like 2D, 3D correspondence. The lastly, there's another stage Network which solves the TMP problem to get a 60 post. So, that's using 3D and 3D post mons and to end the network for 60 post estimation. So, then let's go back to our metab-based param on manipulation. You can see for perception where we have a lot of great master 60, 60 post estimation problem. We can get good of this post. But now, let's see how we do the planning and the motion planning for

manipulation M called crit. It's a model-based cross planning. So basically, this master takes the 3D model of the objects that have some sampling strategy to plan where the cross should be placed to cause objects. Then for each C, it's outscore. We can get a set of the high score C from this method. So this is sample CS in simulation and verify it and additional work. Other media that's can verify all this in a physics simulator for with I gym simulator. We can SP a lot of objects creers then verify each of these brers. So on the right, you can see the master B and this green box in indates after the F each for indicat two fingerer to objects. So with this kind of approach, given a 3D model, we can get the for the objects. So lastly, once we have out and boss plan, we're have a motion plan out them. For example, we use the open motion planning libr in movix that's integrated with The Zing system. We can plan that.

was to pick up objects. So, this BL in this that's indicate styles. Once we objects and estimate the size, we can have for each object with using that 3D model to avoid other objects. But we can use the plan to the pi. There are also some advanced planners that's going to be considered both CLS and the motion. So, for example, this mg planner. It can try to the double arm motion as well as selecting which class is the best for its cre method that's optimized. For example, in Z, the GRE gr is the one which is selected, but the trajectory is not that Lo. So, in F, you can select another class so can get the better trajectory for acting objects. So, this is a Jo and the motion planner and the C is available online. So, as you can see, I have the perception by using F estimation and motion planning for manipulation. I will show you ex how to do some more complicated tasks using this kind of approach. So, here the first look at the handle. We can have 6 6.

part of the handles with motion planning. We plan, we do a motion plan to open the cabinet. Now the robots will inside the cabinet and detect the sugar can. You can motion of frosting and place the sugar box onto the counter, then double counter and place it. So finally, the can look out the handles, detach them, and K the cabinet again. So this example to show if we come down 6 estimation cement planning, we can achieve this kind of complicated manipulation cost. Okay, so for the second half of the talk, I would like to talk a little bit about benchmarking. As you know, the B challenges the mark through the 6 PO estimation. There are more M evaluated, and the performance is increasing every year. But in go B manipulation, we don't have such benchmarks. Some methods use simulation to benchmark. But then this simulation is to different kind of manipulation in the world. The difficulty is in usability. For example, if we want objects, make sure every time we have the same SC compar between different methods.

you just the object. You don't get a fair comparison for different methods. There are only a few benchmarks they can achieve ability ination. For example, this one Le assembly because every team, every me, will use the same so can achieve ability in this ass TKS. Some other coaches, they do AI tasks, they put aack on the table. So every time you put the thing, you can make sure you have the same of the for testing. So recently we introduced a simple idea. We can get R ofex but still make sure every time you put the objects, we have the same for testing in this. We can have comparison methods. So then let talk about this Benchmark for manipulation. So first all, we can get objects that are very widely used for manipulation. We can provide a six uh 16. You can get online by using the Amazon data in order to generate the syn for testing. We first compute the state of post, how this object is going to place on data like this are State Post for placement. We also test the ability of the so we don't want very

can reach in the simulator. We can compute where point we can put these objects and in the ability of the dev. So then with this setup, we generated 20 SC with 16 Optics. Here are some examples in the G simulation. You can see these objects are with different configurations and different positions. So can use this for testing but idea how to achieve the ability is that we have a reference image every time we place objects. We really see how this object is placed. Then we image in this case our students. It's aligning the object in the work with image but the idea that every time you place object we can have the same for testing. We can different process everything and test then have the same. So after the set have the same with benchmarks three type of grasping method. The first one is the me Bas grasping at the top right. We can use six post estimation for crossing. I will recap what's the me cring pipeline. We do 60 up the post estimation and then can do some.

other planning after that motion planning pick-ups, but in this T, we can see the KCK and the place, the pick-up and place it on the side, on the other side we don't have tactics. We can also do me grazing. The idea is based on some optics, for example, in the Build Challenge, we have this of the instance. It can be used, the M foring or given image. We want something of objects, so we have the RBD camera right for set objects. We get a point those objects. We can do some M forast planning. Let's just take point the objects and gener C. After that, we can do mer planning again to which one of the kind gra and enable the we to pick up and place the objects. So here, the to show the model-based cross planning in the real work. The first estimation is from the gdr mppp, which is the method for last the B challenge. So you can see you found this window. We have the detection, that's a UL detection and the six in the

go. So basically, six to recognize objects and pick up and place objects on the side. See, we have five objects and we try 20 things, that's 100 bu. Then we can't. How many access R in working 100 BS. For more of the grafting in this case, we don't have six propos, but we do segmentation and the motion plan and B planning based on the segmentation. So you can see how this Alm work on the Windows. You see the input image, the deps, the segmentation, and the can Ste pre objects just based on segmentation. If something goes wrong, the student will res set the same. So to make sure every time you test, you have the same POS for other objects. For some objects are more difficult, for example, the CES right a BL object on the table. So compared to other objects like the C box, is relatively easier to. Okay, so these are different Ty of approaches we can use for clasping the other type of pipeline foring. I haven't about.

is we don't need to do any perception. For example, the approach they just input an image, then generate actions to objects. We can also use the theme benchmark to benchmark this kind of approach. For example, DEET, it's a new network. It takes the input text image and then predicts where on the text image is that to the that. Then in this case, the operation gets an input image and output action based on the planning. We can do motion plan picks the that is just down and pick up objects. So here, uh, the text has cring on the B panel. Here you can see the prediction because it's toting objects. So in the future, pl we add moreing for example app to set here. Okay, the because stand cre place. Okay, so with all this effort, we also take the B on the manipulation with benchmarks, different types of model Place model play, and end-to-end adjustment approaches. And I would like to show that actually the model-based one, which is based on TDR and GP, the best.

the over method in GOP last year, uh, plus the cross planning using Prit and motion planning using the Open Motion Planning Library, achieved the top BL pick-up accuracy and the J planning accuracy across success accuracy. Can see here it's one on the and we count how many success. Success means you just pick it up but place success means you want to pick up, move, and place it. So in the future, we like to see post-improve the mation. So lastly, I'd like to show you what are thees. So have the TP, that's very good first estimation, but in the testing, we still see found failure. The first failure is object detection. Use a yellow detector, but sometimes, for example, it cannot detect the Z in the. We set the str5, maybe because theor the below5, that's why cannot detect if that post error, the cannot pick up the objects. In this case, the post of the part Dr is gone, so robots cannot break it up. There's no CR plan error as well, so you can see this is a pipeline even if

you have 100% post-estimation plan for the trial, but overall, we see in 100 BS, we get 66 T Set for this pipeline based on GDP, but the perception errors without 24 perception errors. That means in the real results, a large margin to improve for the detection and for the Post-estimation seven planning errors and its three ex errors. Ex means you doing the new management something WR. That's the that's okay. So, so in conclusion, take that 60 post-estimation can facilitate manipulation. We can do more manipulation using 60, 60 of the post-estimation. The performance of 60 patience may not be saturated yet, although we see it's really hard to push on the VB challenge. But if you can see the performance in the drawaway, we still see a lot of fails on detection, the post errors. You need a very accurate post-estimation in the draw work. That's something to think about. Could there be some testing real work in Stu just on.

the dataset then we really improve the usability of this 60 proc estimation method. So, interest the P challenge with s replica Mark, although it already has things five objects, that's about 400 instances that we can compute this Matrix for post estimating the real. Then we really test all this me in real for post post estimation. Uh, in the end, I'd like to thank all my students who have contributed to the work and thank you so much for inviting me to give the talk. I'm ready to take any questions. Thank you. Question [Music] yeah yeah than question. The question is about have we comped any process Matrix for the Z St. Yes, we did. We comped the ads ad ABS. That's similar to the B Matrix. They have that in the paper as well. Uh, basically, the GDP is the best me you better so far better GD postp and post, but we're happy to all more submissions from the P Channel and see how how the [Music] work. Yeah, the question per into the 60 po, that's definitely you.

need a very accurate 60 po because, like Jonathan already mentioned many times, if you get 100 1 cm or one off the other in post Matrix book is small but may affect the CR SP. So, for example, in the 100 block, we see 24 times that's created due to detection of process the perception figures. So, uh, but if you improve this to more DET or clation FES, we see more po figes the detection for the Z many P of not what for this subject but more process. Thank thanks again. Thank you.

R6D workshop (ICCV23) – Invited talk by Yu Xiang

R6D workshop (ICCV23) – Invited talk by Yu Xiang