Report available from 2019 NSF Speech for Robotics Workshop

With robotics rapidly advancing, more effective human-robot interaction is increasingly needed to realize the full potential of robots for society. While spoken language must be part of the solution, our ability to provide spoken language interaction capabilities is still very limited. The National Science Foundation accordingly convened a workshop, bringing together speech, language, and robotics researchers to discuss what needs to be done.

Chaired by Professor Carol Espy-Wilson (ECE/ISR), the Speech for Robotics Workshop was held October 10–11, 2019 at the Brendan Iribe Center for Computer Science and Engineering on the campus of the University of Maryland in College Park, Md. Other members of the organizing committee included Abeer Alwan, UCLA; Joyce Chai, University of Michigan; Dinesh Manocha (ECE/UMIACS/ISR), University of Maryland; Matthew Marge, U.S. Army Research Laboratory; and Raymond Mooney, University of Texas at Austin. The workshop was co-sponsored by NSF and the Department of Electrical and Computer Engineering and the Institute for Systems Research at the University of Maryland.

The final report from this workshop, Spoken Language Interaction with Robots: Research Issues and Recommendations, is now available for download.

The report identifies key scientific and engineering advances needed to enable effective spoken language interaction with robotics. Recommendations broadly relate to eight general themes.

• Meeting human needs requires addressing new challenges in speech technology and user experience design.

• This requires better models of the social and interactive aspects of language use.

• For robustness, robots need higher-bandwidth communication with users and better handling of uncertainty, including simultaneous consideration of multiple hypotheses and goals.

• More powerful adaptation methods are needed, to enable robots to communicate in new environments, for new tasks, and with diverse user populations, without extensive re-engineering or the collection of massive training data.

• Since robots are embodied, speech should function together with other communication modalities, such as gaze, gesture, posture, and motion.

• Since robots operate in complex environments, speech components need access to rich yet efficient representations of what the robot knows about objects, locations, noise sources, the user, and other humans.

• Since robots operate in real time, so must their speech and language processing components.

• In addition to more research, more work is needed on infrastructure and resources, including shareable software modules and internal interfaces, inexpensive hardware, baseline systems, and diverse corpora.

The workshop participants believe research and development that prioritizes these issues will provide a solid foundation for the creation of speech-capable robots with which humans can easily and effectively work.

Download Spoken Language Interaction with Robots: Research Issues and Recommendations, the 2019 NSF Speech for Robotics Workshop Report.

Published November 16, 2020