A framework for low-latency, LLM-driven multimodal interaction on the Pepper Robot
Loading...
Author (Corporation)
Publication date
2026
Type of student thesis
Course of study
Collections
Type
04B - Conference paper
Editors
Editor (Corporation)
Supervisor
Parent work
HRI '26. Proceedings of the 21st ACM/IEEE International Conference on Human-Robot Interaction
Special issue
DOI of the original publication
Link
Related research data
Series
Series number
Volume
Issue / Number
Pages / Duration
1298-1302
Patent number
Publisher / Publishing institution
Association for Computing Machinery
Place of publication / Event location
Edinburgh
Edition
Version
Programming language
Assignee
Practice partner / Client
Abstract
Despite recent advances in integrating Large Language Models (LLMs) into social robotics, two weaknesses persist. First, existing implementations on platforms like Pepper often rely on cascaded Speech-to-Text (STT)→LLM→Text-to-Speech (TTS) pipelines, resulting in high latency and the loss of paralinguistic information. Second, most implementations fail to fully leverage the LLM’s capabilities for multimodal perception and agentic control. We present an open-source Android framework for the Pepper robot that addresses these limitations through two key innovations. First, we integrate end-to-end Speech-to-Speech (S2S) models to achieve low-latency interaction while preserving paralinguistic cues and enabling adaptive intonation. Second, we implement extensive Function Calling capabilities that elevate the LLM to an agentic planner, orchestrating robot actions (navigation, gaze control, tablet interaction) and integrating diverse multimodal feedback (vision, touch, system state). The framework runs on the robot’s tablet but can also be built to run on regular Android smartphones or tablets, decoupling development from robot hardware. This work provides the HRI community with a practical, extensible platform for exploring advanced LLM-driven embodied interaction.
Keywords
Subject (DDC)
Event
HRI '26. 21st ACM/IEEE International Conference on Human-Robot Interaction
Exhibition start date
Exhibition end date
Conference start date
16.03.2026
Conference end date
19.03.2026
Date of the last check
ISBN
979-8-4007-2128-1
ISSN
Language
English
Created during FHNW affiliation
Yes
Strategic action fields FHNW
Publication status
Published
Review
not peer-reviewed
Open access category
Gold
License
Citation
Studerus, E., Zhong, J., & Vonschallen, S. (2026). A framework for low-latency, LLM-driven multimodal interaction on the Pepper Robot. HRI ’26. Proceedings of the 21st ACM/IEEE International Conference on Human-Robot Interaction, 1298–1302. https://doi.org/10.1145/3757279.3788808