Skip to main navigation Skip to search Skip to main content

User-Centric Spatial Referring for Tabletop RobotManipulation with MR Grounding and LLMs

  • Tenma Uchiyama
  • , Yukitsugu Okamura
  • , Alexander Plopski
  • , Maki Sugimoto

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Interpreting perspective-dependent spatial referring expressions is a key challenge for intuitive human-robot interaction, as users naturally communicate in egocentric terms. We present an interactive spatial grounding system that integrates mixed reality (MR) sensing with LLM-based reasoning to translate free-form spatial language into actionable robot targets. Using a unified table-centric representation, the backend computes viewpoint- and robot-relative spatial features and ranks candidate targets under underspecified constraints. We further apply confirmation-gated execution, which visualizes the current target hypothesis in MR and triggers robot motion only after explicit user confirmation. Overall, this demo focuses on enabling robots to understand user-centric, perspective-dependent spatial references by grounding them in MR for tabletop manipulation.

Original languageEnglish
Title of host publicationAHs 2026 - Augmented Humans 2026
EditorsKouta Minamizawa, Shunichi Kasahara, Paul Strohmeier, Yomna Abdelrahman, Maki Sugimoto, Matthias Hoppe, Misha Sra, Nadine Wagener, Steeven Villa, Takefumi Hiraki
PublisherAssociation for Computing Machinery, Inc
Pages748-750
Number of pages3
ISBN (Electronic)9798400723513
DOIs
Publication statusPublished - 2026 May 8
EventInternational Conference on Augmented Human Technologies and Interaction, AHs 2026 - Okinawa, Japan
Duration: 2026 Mar 16 → 2026 Mar 19

Publication series

NameAHs 2026 - Augmented Humans 2026

Conference

ConferenceInternational Conference on Augmented Human Technologies and Interaction, AHs 2026
Country/TerritoryJapan
CityOkinawa
Period26/3/16 → 26/3/19

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

  1. SDG 3 - Good Health and Well-being
    SDG 3 Good Health and Well-being

Keywords

  • Augmented Humans
  • Human-robot interaction
  • Large Language Models
  • Mixed Reality
  • Spatial Interaction

ASJC Scopus subject areas

  • Artificial Intelligence
  • Automotive Engineering
  • Health Informatics
  • Signal Processing

Fingerprint

Dive into the research topics of 'User-Centric Spatial Referring for Tabletop RobotManipulation with MR Grounding and LLMs'. Together they form a unique fingerprint.

Cite this