Okay, it's a good pleasure to be here. Thanks for the wonderful workshop. I'm going to talk about connecting 6D pose with robot manipulation.

So, uh, let's start with model 6D pose estimation as we're working on this problem here. Given an input image, suppose the image is taken from a camera, and here model-based means that we have the 3D model of these objects.

And the setup is that we have a coordinate frame from the objects.

Now the goal is to estimate the location, translation, between the object coordinate frame to the camera frame. So translation means that where the object is in the camera frame, and orientation means that what is the rotation from the object coordinates to the camera coordinates.

So what can we get from the 6D pose estimation? We can know two types of information. The first is we know the object position in the camera, that is the 3D.

We also know what's the orientation in the camera. So we can visualize this information by using input image. Suppose we can estimate the object 6D pose. We can have the point cloud RGB-D camera.

Then we draw this XYZ axis in terms of the RGB color in the in the axis, right? We know where the position, orientation of these objects.

If you have the 3D model of this object, you have to render this model and overlay with the input image.

You can project the 3D model to the input image. So from 6D pose estimation, we can get this information about objects.

Now let's see what's the applications. First of all, we can use 6D pose estimation for robot manipulation. For example, in this case, from the lab previously, we have used it to pick up objects, Y objects, using RGB-D camera.

Right? In this case, we can estimate the pose of the CAD models or the power on the objects or we can use pose estimation to do some detection of in the case of the industry objects.

But today, I'm going to focus more on global manipulation.

So in the traditional model-based manipulation, we have this perception, planning, and control pipeline. In perception, we can focus on the 6D pose estimation problem.

As before, we can detect objects, estimate their poses in 3D. But in planning, we can tackle this class planning and the motion planning problem.

For example, I can set up the planning thing using the 3D model object, then plan the trajectory of the robot to pick up objects.

So finally, in the control stage, the robot just needs to follow the planned trajectory and grasp objects. And in this case, the Franka arm will pick up the picture based on the planning.

So let's start with the perception problem, the 6D pose. In the traditional way of doing 6D, we have two types of methods.

The first one is a feature-based, feature matching-based approach. This approach basically builds a 3D model of texture features. Then you can get a 2D-3D correspondence.

From that, you can solve PnP to get the 6D pose. Or template matching-based methods, if you have a 3D model, render an image of the object model, you can compare this template to the input images.

So it's interesting to know that all the deep learning-based approaches still take inspiration from this traditional 6D pose estimation method.

So next, I will quickly talk about some previous work I have done for 6D pose.

In ECCV 2018, we introduced the first end-to-end 6D pose estimation network called PoseNet.

for input image and then output six multiple objects in the input image, multiple objects, symmetric objects, and the glue different objects.

So this is the first end-to-end post network. It gets an input image. We can use a feature extractor to get some image features. The first branch of the network tries to segment objects.

We get the semantic mask. For the second branch, we ask each pixel to predict where the center of the objects are.

So there are center directions. Each pixel predicts the direction to the center, and we also predict the depth of the objects. That's the depth of the translation.

By using this prediction with center direction, we can do a Hough voting in the network. So a Hough voting layer in the network so it can detect objects and estimate object center.

That's Hough for occlusion. So lastly, suppose we can detect a network, we can just have a sub-network to predict the rotation of the objects.

That's the rotation, 3D rotation, different parts. But by combining all the branches, you can get the 6D pose of objects.

So here's some visualization. For our work, you can detect these wild objects. You can see the segmentation, the bounding box, the 6D pose in terms of XYZ coordinates and the overlay rendering.

You can see the object is still uh, jittering because this is a one-stage prediction network. The regression of the pose is not that great.

So later on, we introduced the D, that's an iterative refinement network that is able to get more accurate pose. But the idea of D is given and compare, as most of previous work has talked about.

The input image, we have the initial pose estimation that can come from any pose estimation method.

We have the 3D model where we can render the 3D model with a given image. Then the network which takes the rendered image and an image and get a data pose.

With a data pose, can add it to the uh original pose to improve it, and this performs the E requirements for pose one and then again input to a network and another pose, then we define the pose, right?

So this iterative process can iter- uh improve the pose.

And according to a report from 2020, the best method is also using this F requirement method in terms of numbers. And you'll be video, you can see that P can fit person L marching, especially in the RGB-D case.

It can be IP requirements. So in the world, you can see that the DM get a very stable pose estimation.

That's due to the inter-requirement network. However, as this approach, it's estimate a single pose estimation.

There's no uncertain or distribution pose. So what we introduce is the pose P. It's a Z black, the future for six WE post tracking.

The idea is to apply a filter to the 60 post demand so we can get a temporal estimation for post tracking as well as certain estimation.

But in Pose RPF, we have a particle, and then the particle consists of two parts: the 3D translation and the orientation distribution.

Given the 3D translation, we can localize the objects in the image. We can get the object ROI. But to estimate the distribution orientation distribution, we can compare the ROI and use it into an incorporated computer feature for the ROI.

Then we can have a lookup by gendering the 3D model into a large number of template images.

So we can compare the all feature to the pose to get a distribution of each orientation of the of the input. And this is multi-modulated by my work on the linking implicit imputation of the rotations.

So the benefit of Pose P is we can do post tracking. So in this video, you can see the camera is moving, so it can track all the 6D poses of all these objects.

But the video here is with SDF based depth requirements.

Okay, so lastly I'd like to mention the GD-Net. That's the overall best method in BOP Challenge last year. So this is a direct network for pose estimation that's got a very great accuracy for the GDR.

And that it's taking input image, then object detector, YOLO and YOLO object detection gets the object ROI.

Then the middle stage of the network predicts for each pixel what the 3D coordinates of the object, what a surface normal of the 3D model of that pixel, as well as the mask.

Then by using this method of the network, we can get, they can get like 2D- 3D correspondence.

Then lastly there's another stage network which solves the PnP problem, to get a 6D pose. So that's using 3D and 3D pose estimation to end the network for 6D pose estimation.

So, uh, then let's go back to our method based grasp on manipulation. You can see for perception where we have a lot of great methods for 6D, 6D pose estimation problem, we can get good of this pose.

But now let's see how we do the planning and the motion planning for manipulation method called G-CRIT.

It's a model-based grasp planning. Uh, so basically this method takes the 3D model of the objects that have some sampling strategy to plan where the grasp should be placed to grasp objects.

Then for each grasp it's outputting a score.

We can get a set of high-score grasps from this method. So this is sample grasps in simulation and verify it and additional work, other media that can verify all this in a physics simulator.

For with Isaac Gym simulator, we can spawn a lot of objects, grippers, then verify each of these grasps.

So on the right you can see the master B and this green box indicates after the force closure indicate two fingered grippers to grasp objects.

So with this kind of approach, given a 3D model, we can get the grasps for the objects. So lastly, once we have our grasp plan, we'll have a motion plan out of them.

For example, we use the Open Motion Planning Library in MoveIt, that's integrated with the ROS system.

We can plan the motion to pick up objects. So this block in this that's indicates obstacles. Once we grasp objects and estimate the 6D pose, we can have for each objects with using that 3D model to avoid other objects, but we can use the plan to the pick.

There are also some advanced planners that's going to be considered both grasps and the motion. So for example, this MG planner, it can try to the double arm motion as well as selecting which grasp is the best for it's a discrete method that's optimized.

For example, in Z, the green grasp is the one which is selected, but the trajectory is not that good. So in F you can select another grasp so you can get the better trajectory for grasping objects.

So this is a joint grasp and the motion planner and the code is available online.

So as you can see, I have the perception by using pose estimation and motion planning for manipulation. I will show you example how to do some more complicated tasks using this kind of approach.

So here, the first, look at the handle, we can have 6D pose of the handles with motion planning.

We plan, we do a motion plan to open the cabinet. Now the robot will inside the cabinet.

and detect the sugar. You can motion planning for grasping and place the sugar box onto the counter. Then double check and place it.

So finally, the can, look out the handles, detach them, and close the cabinet again. So this example to show if we combine down 6D estimation, semantic planning, we can achieve this kind of complicated manipulation task.

Uh, okay, so for the second half of the talk, I would like to talk a little bit about benchmarking.

As you know, the BOP challenges, the benchmark for the 6D pose estimation, there are more and more methods evaluated and the performance is increasing every year.

But in real manipulation, we don't have such benchmarks.

Some methods, they use simulation to benchmark, but then this simulation is too different compared to real manipulation in the world. The difficulty is in repeatability.

For example, if we want objects, make sure every time we have the same setup to compare between different methods, if you just move the object, you don't get a fair comparison for different methods.

Uh, there are only a few benchmarks they can achieve repeatability in manipulation. For example, this one, LEGO assembly, because every team, every method will use the same setup, so it can achieve repeatability in this assembly task.

Some other approaches, they do AR tags. They put AR tags on the table so every time you put the thing, you can make sure you have the same setup for testing.

So recently, we introduced a simple idea. We can get rid of AR tags, but still make sure every time you put the objects, we have the same setup for testing.

In this way, we can have a fair comparison of methods. So then let's talk about this benchmark for manipulation.

So first of all, we can get objects that are very widely used for manipulation. We can provide a 6D uh, 16. You can get online by using the Amazon data in order to generate the synthetic for testing.

We first compute the state of pose, how this object is going to place on data like this, our state pose for placement.

We also test the graspability of the object, so we don't want very... we can reach in the simulator, we can compute where point we can put this objects and in the graspability of the object.

So then with this setup, we generated 20 scenes with 16 objects. Here are some examples in the GA simulation. You can see these objects are with different configuration and different pose.

So you can use this for testing. But the idea how to achieve the graspability is that we have a reference image.

Every time we place objects, we really see how this object is placed, the real image. In this case, our student is aligning the object in the work with image.

But the idea that every time you place object, we can have the same for testing, we can different process everything and test then have the same.

So after the set, have the same with benchmarks three type of grasping method. The first one is the method-based grasping.

At the top right, we can use 6D pose estimation for grasping. I will recap what's the method grasping pipeline. We do 6D up the pose estimation and then can do some other planning after that, motion planning, pick up.

But in this task, we can see the pick and the place, the pick up and place it on the side.

On the other side, we don't have tactics. We can also do method grasping. The idea is based on some optics.

For example, in the Build Challenge, we have this of the instance. It can be used the method for grasping, or given image, we want some grasping of objects.

So we have the RGB-D camera, right, for set objects, we get a point those objects, we can do some affordance planning.

Let's just detect point the objects and generate grasps. After that, we can do motion planning again, to which one of the good grasps and enable the robot to pick up and place the objects.

So here is the video to show the model based grasp planning in the real world.

And the pose estimation is from the GDR-Net++, which is the method for last year's BOP challenge.

So you can see, you found this window, we have the detection, that's a YOLO detection, and the 6D pose. So basically 6D pose to recognize objects and pick up and place objects on the side.

You see we have five objects and we try 20 things, that's 100 trials. Then we count how many successful rates in picking 100 objects.

Uh for more of the grasping, in this case we don't have 6D pose, but we do segmentation and the motion plan and grasp planning based on the segmentation.

So you can see how this algorithm works on the Windows.

R6D workshop (ICCV23) – Invited talk by Yu Xiang

BOP: Benchmark for 6D Object Pose Estimation

6D pose estimation is crucial for robotic manipulation, allowing robots to understand object position and orientation for tasks like grasping and complex manipulation sequences.

Despite advancements in 6D pose estimation, real-world robotic manipulation still faces significant challenges, particularly in object detection and achieving the extreme accuracy required for successful grasping and placement.

Benchmarks for robotic manipulation are essential but difficult to create due to repeatability issues; innovative approaches like using reference images can help standardize testing for fair comparisons across different methods.

What is the core problem that 6D pose estimation aims to solve, and what two types of information can be derived from it?

Can you describe the traditional pipeline for model-based robot manipulation, detailing the role of 6D pose estimation within it?

What are the main differences and advantages of the PoseNet, Pose DPM, and Pose RPF methods for 6D pose estimation, as discussed in the presentation?

How does the GD-Net approach for 6D pose estimation combine object detection, 3D coordinate prediction, and the PnP problem?

What challenges does the speaker identify in benchmarking robot manipulation methods, and how does their proposed benchmark attempt to address these challenges, particularly concerning repeatability?

R6D workshop (ICCV23) – Invited talk by Yu Xiang

Scripts