Nasa R2 humanoid robot relies on Prosilica GigE GC2450C and Halcon to "see"

Go to Prosilica GC2450
Go to GigE Cameras
Go to AVT
Go to Halcon software

Playing a key role in real-time vision recognition: humanoid robot relies on two
AVT Prosilica GC2450C cameras and MVTec Halcon software for stereoscopic vision

.	Built in partnership with General Motors and the Oceaneering Space Systems of Houston, NASA’s Robonaut (R2) is the second generation of highly-dexterous humanoid robot designed to work alongside humans and execute simple, repetitive or dangerous tasks on Earth or on a space station. Anthropomorphic R2, is a fully modular, highly dexterous 300 pound robot that consists of a head and a torso with two arms and two hands.
R2 showing twin Prosilica GC2450C mounted in situ	Click to watch the video
The robot's many technological improvements, from its first iteration, include an improved range of sensors that features two Prosilica GC2450 color cameras from Allied Vision Technologies and an Infra-Red Time-of-Flight (TOF) camera. Capable of speeds more than four times faster than the R1 version, R2 features a total of 350 sensors (for tactile, force, position, range-finding and vision sensing) and 38 power PC processors enabling it to perform functions such as object recognition and manipulation.

How it all works
R2’s advanced machine vision capabilities include two Prosilica GC digital cameras which are housed inside its helmet. (See image above). These provide the robot with stereoscopic vision.

The vision system uses colour, pixel intensity and texture-based segmentation as well as advanced pattern-recognition techniques to extract the necessary information.

To simplify the procedure, the system focuses on certain areas of the image using Region of Interests (ROI). (ROI is a function that allows a certain portion of the available pixels to be read out from the camera resulting in a much faster frame rate and less data to be processed.) In addition, the TOF sensor data allows the background to be removed in order to focus on the object of interest (tools, boxes, etc…)

Built-in classification techniques within the software are used to perform 3D and pattern-recognition functions in real-time to allow R2 to compute feasible trajectories and decide where to place its hands to execute a set of predetermined tasks, such as opening boxes, autonomously.

The software used by the system is MVtec’s Halcon software.

The future for R2
During stage 1 R2 will initially be hard-mounted and stationed in the Destiny laboratory on board the International Space Station (ISS) where it will be monitored while executing tasks and operations similar to those performed on Earth.

If successful, R2 could move on to stage 2 of the mission and become mobile to perform station maintenance tasks such as vacuuming or cleaning filters but the ultimate goal is to send R2 outside the ISS to perform dangerous EVA (Extra-vehicular activity) tasks during stage 3.