Robots that rethink: A SMU project on self-adaptive embodied AI

(Press-News.org) By Vince Chong

SMU Office of Research Governance & Administration – It might not be ubiquitous just yet but embodied artificial intelligence is slowly but surely cementing its place in the world. Robotic systems equipped with sensors and cameras help with everything from factory assembly to surgery, while autonomous, self-driving cars and drones are science fiction no more.

Despite these advances though, there is a limit to what embodied AI can do in unpredictable, everyday environments like homes or offices. Say, a robotic arm may be programmed within perimeters to make a cup of coffee. But hide the cup in a nearby cupboard and the task breaks down. 

It is “simple but frustrating” observations like these that sparked one of SMU Assistant Professor of Computer Science Zhu Bin’s latest projects, which won an Academic Research Fund (AcRF) Tier 2 grant from the Singapore Ministry of Education (MOE).

“In the area of embodied AI, robots could execute a fixed script perfectly. But the moment something changed, like a cup being misplaced, even the most advanced robot may fail [since it’s not in the coded script],” he told the Office of Research Governance & Administration (ORGA). 

“[Filling] that gap between static intelligence and adaptive reasoning inspired this project.” 

The award-winning research Self-Adaptive Planning with Environmental Awareness for Embodied Agents lays the foundation for the next generation of adaptive embodied systems, the computer science expert said, referring to AI “that can reason about why and how to act, not just what to do.” 

This could then be applied successfully to elder care to help with complex, multi-stepped tasks such as meal preparation, medication reminders, and light household chores. Or in healthcare, where they could be put more effectively to work as rehabilitation or therapy companions to track a patient’s movements and adapting exercises based on fatigue or recovery progress.

This is particularly salient in Singapore where elder care and healthcare are “critical” concerns due to its ageing population, the research notes. It adds that such innovations can help drive “socially responsible advancements while contributing to Singapore’s vision of becoming a global leader in AI and smart technologies.”

“In practical terms, the potential applications are broad and socially meaningful,” Professor Zhu said. 

The project is led by Professor Zhu and involves Assistant Professor Kotaro Hara, also of the School of Computing and Information Systems, SMU, as well as Franklin Li Mingzhe, a doctoral candidate, and Associate Professor Patrick Carrington, both of the Human-Computer Interaction Institute at Carnegie Mellon University in the US. 

Gruelling AI training

At its core, the project tackles three key problems in embodied AI’s current form, Professor Zhu explained.

Firstly, many current systems rely too heavily on large language models (LLMs) trained on general world knowledge, which can lead to “unrealistic or irrelevant plans”. Examples of LLMs include AI-powered applications such as ChatGPT and DeepSeek that are designed to understand and produce human-like text. 

Secondly, the research notes, embodied AI, as it is, lacks awareness of environmental changes, and is hence unable to track surrounding changes, even small ones. Lastly, it is inflexible and incapable of adapting to elements that are out of place vis-à-vis its coded blueprint. 

The project aims to resolve these issues through, among other things, vigorous and detailed training. This includes feeding its AI software with roughly 200 distinct kitchen tasks in video form, each repeated about 300 times. 

In total, the system will analyse and learn from some 60,000 high-quality video sequences aimed at tracking all potential changes in the task environment. As the project notes, existing work in video understanding “often neglects the complexities of tracking objects … in dynamic, interactive environments.”

“By incorporating first-person perspectives and real-time object state tracking, [the project] will provide new insights into building embodied AI systems,” it states.

The project also plans to develop a LLM to allow an embodied agent to refine tasks on the fly based on real-time feedback. In other words, it allows the robot to learn from what it sees and senses to refine its next move on the spot, rather than rigidly following a preset script. This, Professor Zhu added, makes the agent “more flexible, reliable, and responsive in everyday situations.”

Hence, using the earlier coffee-making example, if a robot is unable to find a mug in the expected location, it may then infer through its LLM that mugs are typically stored in cupboards. It will then update its task plan to include new subgoals such as "open cupboard" and "search for mug" before making that cup of coffee. 

As the research notes, the “success of this project could reshape the research landscape in embodied agents.”

Modular, plug-and-play approach

Not surprisingly, Professor Zhu said, his team’s research will be designed in a way that can be directly integrated with future adaptive AI models. This is to avoid falling behind the curve given the rapid pace of technological development.

“Our focus is on general and transferable algorithms, not on a single model snapshot,” he said.

“Rather than competing with every new model, we will design our framework to be modular and plug-and-play. That’s how we remain relevant and resilient in a fast-moving field.”

The academic is also sanguine about the risk of AI misbehaving or going rogue, a concern that has emerged lately among experts. Is he at all worried, particularly with using embodied agents to help physically weaker demographics such as the elderly and infirmed?

Safety, he said, is always a key consideration in any project that “aims to move embodied AI systems towards deployment in the physical world.” The best methods to mitigate concerns are “through human-in-the-loop supervision, rule-based constraints, and extensive real-world testing” before any deployment in sensitive contexts like elder care or healthcare. 

“These ensure that embodied agents act within well-defined boundaries and always under human oversight,” Professor Zhu said.

“Ultimately, the goal is to develop empathetic, environment-aware AI assistants that complement human care and promote safer, more independent living.”

END