Robot Head Software Operation (Vergence Feature Extraction Camera and PTU control)

The control of the robot head consists of 3 windows displaying the left and right camera images and an anaglyph depicting the stereo registration between both cameras while the system verges on a point. In order to operate the system, it is needed to firstly calibrate the actuators (as described in Actuator calibration MATLAB functions provided in the repository in MATLAB/ActuatorCalibration folder). These matlab functions consist of a source code to compile MEX files for controlling the cameras and PTUs and, similarly, for extracting SIFT features on the GPU.

After calibrating the actuators, it is required to update rh_launch/launch/RH_parameters.launch accordingly. In this file, "/RH/usbcam_left usb:XXX,YYY" and "/RH/usbcam_right usb:XXX,YYY" has to be updated by issuing "lsusb" in order to retrieve the port address for each camera. Left and right is determined by looking at the back of the robot head (similar to say if one wants to determine which of our eyes is left or right). In this way, the left and right ports are retrieved correctly. "/PTU/stepsPerPixelX" and "/PTU/stepsPerPixelY" are obtained from calibrating the actuators as above.

"/PTU/port /dev/ttyUSB0" sets the serial port where the PTUs are connected. In UG, we do not have a physical serial port, therefore we have used a usb to serial converter and this is recognised as "/dev/ttyUSB0" under Ubuntu. "/RH/sift_gpu/lib_path" contains the path where the SIFT GPU library is located locally and, finally, "RH/sift_gpu/ratio_max" sets the log-likelihood threshold (match sensitivity) while matching SIFT features while verging the cameras (see Lowe's paper).

To start the vergence demo, there are two possible launch files:

  • rh_launch/launch/vergence_demo.launch (used in UG)
  • rh_launch/launch/vergence_demo_remote.launch (used in CVUT)

Launch them using roslaunch and nodes should configure and start automatically.

Interaction with the robot head is by means of the “Left camera” window. Three operational modes can be established as listed below:

  • Left mouse click on a point in the “Left camera” window: The gaze of the robot head can be controlled by selecting a point on the window (i.e. clicking on a point in the window). The robot will then capture images, extract SIFT features, compute the disparity between cameras and calculate the required actuator steps to bring the cameras into convergence. This sequence continues until the disparity between is minimised and the system is ready to receive another command specified by the user.
  • Right mouse click in the “Left camera” window: This will command the pan and tilt units to return to home and the system will automatically verge on the cameras following the above procedure.
  • Middle mouse click in the “Left camera” window: This mode sets the capture mode of the cameras; e.g. full or low resolution images. Thereafter, the user can select one of the above operational modes.

The operation of the system can be stopped by issuing “Control+C” on every terminal window running a ROS node.

NOTE: if you are using a remote computer for SIFTGPU, please follow this instructions

If there are problems or bugs, please contact Gerardo Aragon