How Do Robots Sense and Understand Their Environment?

[Un article de The Conversation écrit par Stéphane Bazeille – Enseignant chercheur en robotique, Université de Haute-Alsace (UHA)]

Robots intended for the general public are increasingly present in our daily lives, but today they remain quite close to wheeled or flying platforms (vacuum cleaners, lawn mowers, drones for example). The industry is equipped with specialized arms for assembly and handling. These industrial and everyday robots have one thing in common: they have few sensors allowing them to perceive the world. This makes them easier to manage, but greatly limits their capabilities.

However, in recent years we have observed the emergence of much more complex robots. Humanoid robots, such as those from Boston Dynamics or more recently Tesla, are the most striking examples. These more advanced robots remain research robots today: they can do many more things, but their programming requires many more sensors, as is the case with Atlas, the robot from Boston Dynamics, in this video.

%iframe_0%

If their mechanical capabilities are increasingly spectacular, the main difficulty today is to give robots perceptual capabilities to be able to easily interact with their environment.

Indeed, it is our perception of the environment that allows us, humans, to recognize and locate objects in order to catch them, to estimate their trajectories to anticipate their positions, to move from a point to another by avoiding obstacles for example.

Perception can today be based on different sensors which measure different physical quantities such as light flux, distance, accelerations or rotations.

In humans, the inner ear perceives the position and orientation of the head and allows them to maintain their balance, while drones or humanoid robots maintain their balance by measuring the accelerations and rotational speeds of their bodies, measured at very high frequency thanks to an inertial unit. Vacuum cleaners, for their part, avoid obstacles using distance sensors that allow them to build maps of their environment.

All these tasks (balance, localization, analysis of the environment) are essential to improve the autonomy of robots, but equipping a robot with a perception system is a considerable job: you have to acquire the data, process it, and take a decision on the behavior of the robot based on this information.

From eyes to brain, from cameras to on-board computer

Each sensor returns more or less “high-level” information to the on-board computer, that is to say requiring more or less processing subsequently to extract meaning. The richest sensors, but also the most complex to use, are those developed for vision.

Humans have a highly developed visual system trained since early childhood to recognize, locate, measure and estimate movements: our eye provides a raw image, but our brain knows how to interpret it.

Similarly, the data encoding images from traditional cameras is very “low level”: a camera simply records an ordered list of pixels that correspond to the amount of light received on a small photosensitive surface element of the sensor, to which potentially adds information about color (wavelength) — like our eye.

For comparison, simple “high level” information usable by a robot would be for example:

“There is a white object on the table and it is located at a distance of 100 millimeters in x and 20 millimeters in y from the corner of the table. »

Providing this type of information to a robot in real time is possible today with the processing capabilities of computers. If we take the example of industrial robotic arms, they are today sold without a perception system and cannot catch objects if their positions and orientations have been given to the robot during programming. To be able to catch objects, regardless of how they are arranged, you must give the robot the opportunity to see the object.

This is possible today thanks to “intelligent” cameras, that is to say, which incorporate a computer and image processing libraries to transmit directly usable information to the robots.

Thus equipped, the robot can grab an object, but this time, regardless of how it is arranged.

%iframe_2%

Allow robots to see in 3D

Another challenge for robots is moving in a changing environment. In a crowd, for example, humans constantly estimate their movements and construct a map of the surroundings to determine free zones and occupied zones in order to estimate a trajectory leading to their destination.

On a robot, with only a monocular camera, doing simultaneous localization and mapping is a very complex problem. And the result obtained is an approximate result because we have a problem of “scale ambiguity”, that is to say that the movements are well estimated, but the distances are accurate within a scale factor. To remove this ambiguity of scale, “multi-views” are needed — two eyes in our case, or two cameras.

Integrating two eyes on a robot is tricky because with two sensors, there is twice as much information to process, synchronize and calibrate to obtain the precise position of one camera relative to another.

Thanks to the evolution of vision sensors, we can now see in 3D with a single camera light-field. These cameras are a little special: using a matrix of micro-lenses located between the sensor and the lens, they capture the light intensity of a scene like on a conventional device, but also the direction of arrival of the light rays . This makes it possible in particular to find depth, therefore 3D, with a single image.

A small robot equipped with this type of camera, for example, can obtain a location and a metric map consistent with reality (by removing scale ambiguity). This is the work that we carry out today at the IRIMAS laboratory, particularly as part of a thesis.