Mediating Perception and Action towards Situated Human-Robot Dialogue

A new generation of robots have emerged in recent years which serve as humans' assistants and companions. However, due to significantly mismatched capabilities in perception and action between humans and robots, natural language based communication becomes difficult. First, the robot's representation of its perceived world and its action space are often continuous and numerical in nature. But human language is discrete and symbolic. For the robot to understand human language and thus take corresponding actions, it needs to first ground the meanings of human language to its own sensorimotor representations. Second, the robot may not have complete knowledge about the shared environment and the joint task. It may not be able to connect human language to its own representations. It is important for the robot to continuously acquire new knowledge through interaction with humans and the environment.

To address these challenges, we have been working on mechanisms to mediating the gaps of perception and action between humans and robots. To mediate perceptual differences, we have developed graph-based approaches to support collaborative dialogue for referential grounding. We have also developed collaborative models for referring expression generation that takes into account of perceptual differences. To address the gap of action, we are exploring how to make a robot (a robotic arm) learn new high-level actions through natural language dialogue. In this talk, I will give an introduction to this line of research and discuss our approaches and empirical results.

Joyce Chai is a Professor in the Department of Computer Science and Engineering at Michigan State University. She received a Ph.D. in Computer Science from Duke University in 1998. Prior to joining MSU in 2003, she was a Research Staff Member at IBM T. J. Watson Research Center. Her research interests include natural language processing, situated dialogue agents, information extraction and retrieval, and intelligent user interfaces. She has served as a Program Co-chair for the Annual Meeting of Special Interest Group on Dialogue and Discourse (SIGDIAL) in 2011 and a Program Co-chair for the ACM International Conference on Intelligent User Interfaces in 2014. She received a National Science Foundation Career Award in 2004 and the Best Long Paper Award from the Annual Meeting of Association of Computational Linguistics (ACL) in 2010.