We are interested in exploring the interplay of hand-eye coordination for dynamic grasping tasks where objects to be grasped are moving. Coordination between an organism's sensing modalities and motor control system is a hallmark of intelligent behavior, and we are pursuing the goal of building an integrated sensing and actuation system that can operate in dynamic as opposed to static environments. The system we are building is a multi-sensor system that integrates work in real-time vision, robotic arm control and stable grasping of objects. Our first attempts at this have resulted in a system that can track and stably grasp a moving model train in real-time. (See figures and video below.)
The algorithms we have developed are quite general and applicable to a variety of complex robotic tasks that require visual feedback for arm and hand control. Currently, we are extending this work to tracking in a full 3-D space, and instituting 2 new control algorithms that employ multiple moving object detection and a wrist mounted camera for finer tracking and grasping. This work is important in showing 1) that the current level of computational devices for vision and real-time control are sufficient for dynamic tasks that include moving objects, 2) optic-flow is robust enough to be computed in real-time for stereo matching and 3) it can define strategies for robotic grasping with visual feedback that may be motivated by human arm movement studies.
Robot arm picking up moving train.
Click
for an mpeg video of our experiment.
WARNING!!! the video is 4880128 bytes long so you may have to wait
a while...