Abstract
Interpreting perspective-dependent spatial referring expressions is a key challenge for intuitive human-robot interaction, as users naturally communicate in egocentric terms. We present an interactive spatial grounding system that integrates mixed reality (MR) sensing with LLM-based reasoning to translate free-form spatial language into actionable robot targets. Using a unified table-centric representation, the backend computes viewpoint- and robot-relative spatial features and ranks candidate targets under underspecified constraints. We further apply confirmation-gated execution, which visualizes the current target hypothesis in MR and triggers robot motion only after explicit user confirmation. Overall, this demo focuses on enabling robots to understand user-centric, perspective-dependent spatial references by grounding them in MR for tabletop manipulation.
| Original language | English |
|---|---|
| Title of host publication | AHs 2026 - Augmented Humans 2026 |
| Editors | Kouta Minamizawa, Shunichi Kasahara, Paul Strohmeier, Yomna Abdelrahman, Maki Sugimoto, Matthias Hoppe, Misha Sra, Nadine Wagener, Steeven Villa, Takefumi Hiraki |
| Publisher | Association for Computing Machinery, Inc |
| Pages | 748-750 |
| Number of pages | 3 |
| ISBN (Electronic) | 9798400723513 |
| DOIs | |
| Publication status | Published - 2026 May 8 |
| Event | International Conference on Augmented Human Technologies and Interaction, AHs 2026 - Okinawa, Japan Duration: 2026 Mar 16 → 2026 Mar 19 |
Publication series
| Name | AHs 2026 - Augmented Humans 2026 |
|---|
Conference
| Conference | International Conference on Augmented Human Technologies and Interaction, AHs 2026 |
|---|---|
| Country/Territory | Japan |
| City | Okinawa |
| Period | 26/3/16 → 26/3/19 |
UN SDGs
This output contributes to the following UN Sustainable Development Goals (SDGs)
-
SDG 3 Good Health and Well-being
Keywords
- Augmented Humans
- Human-robot interaction
- Large Language Models
- Mixed Reality
- Spatial Interaction
ASJC Scopus subject areas
- Artificial Intelligence
- Automotive Engineering
- Health Informatics
- Signal Processing
Fingerprint
Dive into the research topics of 'User-Centric Spatial Referring for Tabletop RobotManipulation with MR Grounding and LLMs'. Together they form a unique fingerprint.Cite this
- APA
- Standard
- Harvard
- Vancouver
- Author
- BIBTEX
- RIS