A robot spots and fixes wrong object descriptions in its memory, MIT says

MIT researchers built UQ-DAAAM, which checks a robot's stored object descriptions against other camera views and corrects those that look doubtful. The results are the researchers' own, from a paper to be presented at a conference.

Share
A robot spots and fixes wrong object descriptions in its memory, MIT says
Illustration: AI-generated

A robot that mistakes a grey chimney for a squirrel can now measure how unsure it is about what it saw, and correct the description as it moves, MIT researchers say. The memory system is called UQ-DAAAM and is described on MIT's news site.

The team's earlier system, DAAAM, uses a vision-language model, which turns images into text descriptions, to describe objects seen by a robot's cameras. It stores the descriptions in a 3D map the robot can use later to answer questions.

Errors arise when the robot sees an object only partly, from far away, or in a blurry or poorly lit image.

MIT gives an example: a robot asked in an emergency to fetch a wall-mounted defibrillator may not find it if its memory has labelled it a fire extinguisher. Harry Zhang, a co-lead author and MIT graduate student, says that without a way to correct itself, a wrong memory would persist.

UQ-DAAAM first calculates how confident the model is in the words it used. It then checks whether that description matches captions of the same object taken from other angles.

Luca Carlone, an associate professor in MIT's Department of Aeronautics and Astronautics and director of the MIT SPARK Laboratory, says the two checks are combined into a single number. If it rises above a set limit, the system looks at more images of the object and rewrites the caption.

A simple rule picks the extra views that will reduce the robot's uncertainty. A large language model, which writes text, then merges the chosen captions into one description, giving more weight to newer captions and to those with less ambiguity.

In MIT's example, the system first wrongly calls the object a squirrel. As the robot keeps moving, it describes the object, in MIT's words, as “a dark gray, cylindrical chimney with a slightly tapered top and a smooth surface.”

In their tests, the researchers report that the refinement step resolved ambiguities for about 92% of objects in the robot's environment, using two extra captions per object. They say it made descriptions 5% more accurate than the state of the art.

Compared with baseline methods, UQ-DAAAM also gave more accurate answers about which objects the robot saw, where it saw them and when it last saw them.

The paper, to be presented at the Conference on Neural Information Processing Systems, is described in MIT's release, also posted on EurekAlert. Zhang and Nicolas Gorlo, also an MIT graduate student, are co-lead authors, with Carlone as senior author.

Next, the researchers want to use UQ-DAAAM to guide other types of robots, such as mobile robotic manipulators. They are also studying whether it can find areas of high uncertainty that a robot could revisit.

Sources