In the office in Mit's Computer Science and Artificial Intelligence Laboratory (CSAIL), a soft robotic hand cautiously rolls his fingers to capture a small object. The intriguing part is not a mechanical project or built -in sensors – in fact, the hand does not contain any. Instead, the whole system is based on one camera that observes the robot movements and uses these visual data to control it.
This ability comes from a new system developed by CSAIL scientists, offering a different view on robotic control. Instead of using hand -designed models or complex sensor matters, it allows robots to find out how their bodies react to control commands, only through seeing. The approach, called the neural areas of Jacobian (NJF), gives robots a kind of bodily self -awareness. Some Open work paper was published in Nature June 25.
“This work indicates the transition from programming robots to didactic works,” says Sizhe Lester Li, PhD student in the field of electrical engineering and computer science, CSAIL partner and main work researcher. “Today, many works of robotics require extensive engineering and coding. In the future, we imagine showing a robot what to do, and allowing him to autonomously achieve the goal.”
Motivation results from a simple but powerful freezing: the main barrier at an affordable price, flexible robotics is not equipment – this is a control of the ability that can be achieved in many ways. Traditional robots are built so that they are stiff and rich in sensors, which facilitates the construction of a digital twin, a precise mathematical replica used for control. But when the robot is soft, deformed or irregular shape, these assumptions fall apart. Instead of forcing the work to match our models, NJF reverses the script – giving the robots the opportunity to explore its own internal model from observation.
Look and learn
This separation of modeling and design design can significantly expand the design space for robotics. In soft and BIO inspired works, designers often deposited sensors or strengthen parts of the structure so that modeling is possible. NJF raises this limitation. The system does not need on -board sensors or design corrections to allow control. Designers are more free in discovering unconventional, unlimited morphology, without worrying about whether they will be able to model or control them.
“Think about how you learn to control your fingers: you move, watch, adapt,” says Li. “This is what our system does. He experiments with random actions and numbers that control, move which parts of the robot.”
The system turned out to be solid in various types of robots. The team tested NJF on a soft -robotic pneumatic hand capable of pinching and gripping, a stiff Allegro hand, a robotic arm with a 3D print, and even a rotary platform without built -in sensors. In each case, the system has learned both the shape of the robot and how it reacted to control signals, only by view and random movement.
Scientists see potential far beyond the laboratory. Robots equipped with NJF can one day perform agricultural tasks with a centimeter level accuracy, operate on construction sites without complex sensor matters or move in dynamic environments where traditional methods are distributed.
At the base of NJF there is a neural network that reflects two intertwined aspects of the incarnation of the robot: its three -dimensional geometry and sensitivity to control inputs. The system is based on the fields of neural glow (NERF), techniques that reconstruct 3D scenes from images by reproducing spatial coordinates to the value of colors and density. NJF expands this approach, learning not only the shape of the robot, but also the fields of Jacobian, a function that predicts how any point on the body of the robot moves in response to motor commands.
To train a model, the robot makes random movements, while many cameras record the results. It does not require any human supervision or earlier knowledge of the robot structure – the system simply causes a relationship between control signals and movement, observing.
After completing the training, the robot needs only one monocular camera for controlling a closed loop in real time, operating at about 12 Hertz. This allows him to constantly observe, plan and act reactionly. This speed makes NJF more profitable than many physics -based simulators for soft robots, which are often too intense for real -time use.
In early simulations, even straight fingers and 2D sliders were able to learn this mapping using only a few examples. Modeling how specific points deform or move in response to the action, NJF builds a dense control map. This internal model allows him to generalize in the body's body, even when the data is noisy or incomplete.
“It is really interesting that the system stands out itself which engines control which parts of the robot,” says Li. “This is not programmed – it emerges naturally through learning, just like a person discovers the buttons on the new device.”
The future is soft
For decades, robotics favored rigid, easily modeled machines – such as industrial weapons found in factories – because their properties simplify control. But the field is heading towards soft, biologically inspired by robots that can smoothly adapt to the real world. Compromise? These robots are more difficult to model.
“Today's robotics often feel out of reach because of expensive sensors and complex programming. Our goal with the neuronous areas of Jacobian is to reduce the barrier, make robotics affordable, flexible and available for more people. Vision is a resistant, reliable sensor,” says the senior author and assistant to the MIT professor Vincent Sitzmann, who leads the group representation. “He opens the door to robots that can operate in a disordered, unstructured environment, from farms to construction sites, without expensive infrastructure.”
“The vision itself can provide tips needed for location and control-elimination of the needs of GPS, external tracking systems or complex on-board sensors. This opens the door to solid, adaptive behavior in unstructured environments, from drones moving in rooms or underground Rus, professor of electrical and computer science and director of CSail. These systems develop internal models of own movement and dynamics, enabling flexible, self -sufficient activity in which traditional location methods fail. “
During NJF training, it currently requires many cameras and must be converted for every robot, scientists already imagine a more available version. In the future, hobbyists could record random robot movements using a phone, just like you made a movie with a rented car before driving away, and use this material to create a control model, without prior knowledge or special equipment.
The system does not generalize in various robots and lacks the strength or feeling of touch, limiting its effectiveness in tasks rich in contact. But the team examines new ways of solving these restrictions: improvement in generalization, occlusion service and extension of the model's ability to reason on longer spatial and time horizons.
“Like people, they develop an intuitive understanding of how their bodies move and react to orders, NJF gives robots so embodied self -awareness through the vision itself,” says Li. “This understanding is the basis for flexible manipulation and control in real environments. Our work essentially reflects a wider trend in robotics: a departure from manual programming of detailed models towards teaching robots through observation and interaction.”
This article accumulated a computer vision and self -sufficient learning from the Sitzmann laboratory and specialist knowledge in soft robots from Rus Lab. Li, Sitzmann and Rus co -author of the article with CSIAIL Annan Zhang SM '22, PhD student in the field of electrical engineering and computer science (ECS); Boyuan Chen, PhD student at ECS; Hanna Matusik, a undergraduate researcher in mechanical engineering; And Chao Liu, Postdoc in a sensible city laboratory in myth.
The research was supported by Solomon Buchsbaum Research Fund through the MIT Research Support Committee, the Presidential Myth of the Scholarship, the National Science Foundation and the Guangju Institute of Science and Technology.
