A Probabilistic Framework for Object Search w/ 6-DOF Pose Estimation
International Journal of Robotics Research, 2010 (submitted)
Dr. Jeremy Ma, Jet Propulsion Laboratory
Dr. Timothy Chung, Naval Postgraduate School
Dr. Joel Burdick, California Institute of Technology

Robustness to Varying Objects and Elevations

The following videos illustrate successful runs with objects of various shapes and colors and at various elevations.

Object Video Link
(right click on image and save as)
Description Size
Object placed ahead of robot on box at level elevation with robot height. This is a straightforward application of the proposed algorithm. More difficult cases will be shown in the videos below.

Shown is the grid-based probability map generated from initial scan of room, the planned path in blue, the actual path in magenta, and the classified stereo points (red for obstacle, green for ground plane) superimposed. Not shown is the costmap populated from classified stereo for obstacle avoidance.
11.7 MB
Object placed to the front left of robot on box at high elevation. Note that as the robot applies the local-search method for 6DOF pose estimation, it goes through a predefined sequence of search patterns covering a search hemisphere. If there exist enough feature correspondences with some area of the scene, the search sequence on the hemisphere is abandoned and the robot centers the camera on the detected object origin.

Shown is the grid-based probability map generated from initial scan of room, the planned path in blue, the actual path in magenta, and the classified stereo points (red for obstacle, green for ground plane) superimposed. Not shown is the costmap populated from classified stereo for obstacle avoidance.
9.4 MB
Object placed to the front left of robot on box at level elevation with the robot. In this example, the object was trained in the upright position yet due to the chosen features (SIFT), the object is still detectable in other orientations.

Shown is the grid-based probability map generated from initial scan of room, the planned path in blue, the actual path in magenta, and the classified stereo points (red for obstacle, green for ground plane) superimposed. Not shown is the costmap populated from classified stereo for obstacle avoidance.
16.1 MB
In this experiment, the object is placed to the far left of robot on box at level elevation with the robot. Note that the local search with 6-DOF pose estimation struggles to localize the object. However, after several attempts and autonomous re-poisitioning of the camera (via pan-tilt capabilities), the object is properly localized.

Shown is the grid-based probability map generated from initial scan of room, the planned path in blue, the actual path in magenta, and the classified stereo points (red for obstacle, green for ground plane) superimposed. Not shown is the costmap populated from classified stereo for obstacle avoidance.
12.8 MB
The object is placed to the far left of robot on a box at a high elevation. The robot is able to find the object quite easily, primarily because of the strong color signature in the color histogram of the yellow mustard bottle. 6-DOF localization via the local search method is shown as well.

Shown is the grid-based probability map generated from initial scan of room, the planned path in blue, the actual path in magenta, and the classified stereo points (red for obstacle, green for ground plane) superimposed. Not shown is the costmap populated from classified stereo for obstacle avoidance.
13.3 MB
The object is placed to the front of the robot at level elevation with the height of the robot. The robot is able to find the object quite easily, primarily because of the strong color signature in the color histogram of the orange clock. 6-DOF localization via the local search method is shown as well.

Shown is the grid-based probability map generated from initial scan of room, the planned path in blue, the actual path in magenta, and the classified stereo points (red for obstacle, green for ground plane) superimposed. Not shown is the costmap populated from classified stereo for obstacle avoidance.
15.1 MB
The object is placed to the front right of the robot at level elevation with the height of the robot. The robot is able to find the object quite easily, primarily because of the strong color signature in the color histogram of the yellow lunch box. 6-DOF localization via the local search method is shown as well.

Shown is the grid-based probability map generated from initial scan of room, the planned path in blue, the actual path in magenta, and the classified stereo points (red for obstacle, green for ground plane) superimposed. Not shown is the costmap populated from classified stereo for obstacle avoidance.
12.0 MB


Robustness to Replanning (after failed object detection/search)

These are some videos of experiments where the robot fails to find the object on the first attempt, yet replans to the next likely location (using the probability map) and successfully detects and localizes the object.

Object Video Link
(right click on image and save as)
Description Size
In this experiment, the robot initially investigates an area of the environment that contains an object with a color histogram distribution that matches that of the desired object. Upon failing to detect the object in that cell location, the robot successfully replans to the next probable location and finds the object there.

Shown is the grid-based probability map generated from initial scan of room, the planned path in blue, the actual path in magenta, and the classified stereo points (red for obstacle, green for ground plane) superimposed. Not shown is the costmap populated from classified stereo for obstacle avoidance.
21.6 MB
In this experiment, the initial setup was an obstacle avoidance test. However, the robot detects the fire hydrant on the wall to have a color distribution which matches that of the desired object (Bob's Big Boy model) and goes there first. Upon failing to detect the object in that cell location, the robot successfully replans to the next probable location and finds the object there.

Shown is the grid-based probability map generated from initial scan of room, the planned path in blue, the actual path in magenta, and the classified stereo points (red for obstacle, green for ground plane) superimposed. Not shown is the costmap populated from classified stereo for obstacle avoidance.
31.0 MB


Robustness to Obstacle Avoidance (costmap generation)

The following videos illustrate successful runs, illustrating costmap generation using stereo. Keep in mind that most objects are placed on pedestals or boxes that often get detected as obstacles. In order to allow the robot to plan a path to a goal location that may contain an obstacle, the A* planner suppresses the cost of cells in a neighborhood about the goal location.

Object Video Link
(right click on image and save as)
Description Size
In this experiment, an obstacle (white box) is placed directly in front of the robot, with the object to be detected placed just beyond the obstacle. This setup requires the robot to maneuver around the obstacle to successfully detect the object.

Shown in the video is the grid-based costmap (10cm x 10cm resolution) generated by using full stereo classified as either ground-plane or not ground-plane.
22.5 MB
In this experiment, the object (Raisin-Bran cereal box) is placed to the far left of the robot, with the object to be detected placed on a stool at level elevation with the robot height. Note that due to the reflectivity of the waxed floor, false stereo returns apear at the fringe of the stereo detection range. Nonetheless, the wall and chair are picked up as obstacles successfully.

Shown in the video is the grid-based costmap (10cm x 10cm resolution) generated by using full stereo classified as either ground-plane or not ground-plane.
21.8 MB