Teaching Robots to Understand the World While Using Fewer Computing Resources
Researchers at the Moscow Institute of Physics and Technology (MIPT) have developed a method that enables large language models to interpret video, track moving objects and their interactions, and predict what will happen next. The approach dramatically reduces computational requirements and could help robots analyze their surroundings more efficiently. The findings were published in the journal Technologies.
As robots move beyond factory floors into unstructured real-world environments, giving them the ability to understand their surroundings requires enormous computing power, making autonomous systems expensive and heavily dependent on cloud infrastructure. Researchers at MIPT have developed an algorithm that allows robots to analyze live video streams, track objects, and anticipate future events with the support of a large language model. Instead of processing every video frame sequentially through a computationally intensive pipeline, the system converts the footage into a compact "event graph." Within this representation, nodes correspond to key objects, while edges capture their spatial and temporal relationships. The approach preserves the underlying semantics of a scene while dramatically reducing computational demands by mimicking key principles of human perception.
Triumph on the Test Platform
The algorithm was successfully tested on a Husky mobile robot equipped with a robotic manipulator. The robot received commands in Russian, such as, "Drive to the table and pick up the object," then executed them by analyzing live video of its surroundings in real time. On the STAR benchmark dataset, the system answered 99% of questions about object interactions correctly and predicted the next action with 97% accuracy. Although the project remains a research prototype rather than a commercial product, the results provide strong evidence that the underlying concept is technically viable.
Significance for the Industry and for Russia
The research sits at the intersection of computer vision, large language models, and edge computing. It represents a practical path toward autonomous robots capable of operating without relying on powerful remote servers. For users, the technology could improve the reliability and affordability of service robots, medical assistants, and domestic helper robots. For Russia, it offers an opportunity to reduce the cost of integrated hardware and software robotic platforms while accelerating the deployment of autonomous systems across manufacturing and logistics. Those goals align with the country's strategy of joining the world's top 25 nations in industrial robot adoption by 2030 through a series of federal development initiatives.
Global Context
Robotics research worldwide is moving in a similar direction, placing increasing emphasis on intelligent autonomy. In 2022, researchers at AIRI introduced depth-estimation algorithms that eliminated the need for expensive LiDAR systems. In 2024, researchers at the V.A. Trapeznikov Institute of Control Sciences of the Russian Academy of Sciences combined language models with sensor data. By 2025, global technology companies, including Google with Gemini Robotics On-Device and Nvidia with the Isaac GR00T N1 platform, had introduced integrated vision-language-action systems. The Russian approach compares favourably with other vision-language-action systems by emphasizing computational efficiency, enabling intelligent processing to run directly on the robot instead of on remote infrastructure.
Prospects and Challenges
Over the next several years, the algorithm is expected to find applications in robots and robotic manipulators with limited onboard computing resources, including warehouse automation, manufacturing inspection, industrial sorting, and the exploration of hazardous environments. Its export potential is primarily long-term, with the most likely commercial product being a software module or software development kit (SDK) for systems integrators in countries seeking cost-effective automation. The principal limitation remains the shortage of annotated datasets that combine textual descriptions with graph-based representations of video scenes. Without substantially expanding these datasets, the model will struggle to compete with commercial alternatives. Additional testing under more complex and dynamic operating conditions will also be required.
The MIPT project represents a transition from robots driven by rigid scripted behaviors to autonomous systems capable of understanding context and selecting actions independently. Its central contribution lies in combining high prediction accuracy with a dramatic reduction in the amount of information that must be processed. Over the next two to three years, researchers expect to expand the training dataset, evaluate the system on additional robotic platforms, and launch pilot projects with industrial partners. If successfully refined, the technology could become a foundational Russian software component for the next generation of autonomous robots, strengthening the country's technological capabilities and software independence.
