Research in the Sociable Agents Group
Our research projects target systems and tools to make machines conversational, cooperative, convergent, and companionable, and to explore these abilities in novel human-machine interaction scenarios.
Adaptive Embodied Communication
with Matthias Weigelt (Univ. Saarbrücken), Thomas Schack (Sport Sciences)
Instructions about sequences of actions are better memorized when offered with appropriate gestures. In this project, a virtual character is used to give instructions with self-generated gestures. The instructions are tailored to the specific needs of the user depending on the user's memory representations as assessed using the SDM-A method. The effect of this adaptive embodied action tutoring on the action learning is investigated.
AMALIS – Adaptive Machine Learning of Interaction Sequences
with Thomas Herrmann (Ambient Intelligence)
Interaction scenarios are full of multivariate sequences of data, e.g., speech, nonverbal behavior streams, data streams from interface operation. This project explores how systems can be enabled to learn such data in highly dynamic environments, such as companion-based interaction with smart rooms, in which unpredictable changes may occur and the systems must re-adjust at many events. We develop models for unsupervised and reinforcement learning of hierarchical structure from sequential data, which not only afford the analysis of sequences but also generation of learned patterns.
Conceptual Motorics
with K. Rohlfing, I. Wachsmuth, F. Joublin; CoR-Lab
A crucial steps in the attempt to build sociable, humanoid robots is to endow them with nonverbal expressivity. This project aims for robots that can derive speech-accompanying hand movements from conceptual to-be-communicated information, e.g., to point to objects being referred to or to illustrate actions currently discussed. The first objective is to enable the humanoid robot ASIMO to flexibly produce synthetic speech along with expressive hand gesture without being limited to a predefined repertoire of motor actions. A second objective is to exploit this flexibility for controlled experiments realized on ASIMO which enable new insights into human perception and understanding of gestural machine behaviors and how to use these in designing artificial communicators. See the project website for more information.
DiaCoSA – Dialogue Coordination for Sociable Agents
Smooth and trouble-free interaction in dialogue is only possible if interlocutors are able to coordinate their actions with each other. Because of this, we think that an understanding of how human coordination devices work is the basis for successful spoken language computer interfaces such as embodied conversational agents or dialogue systems. In this research project we model two important coordination mechanisms that are commonly found in human-human interaction: feedback and adaptation. Read more …
IMoRa – Imitation Mechanisms and Motor Cognition for Social Embodied Agents
This project explores how the brain's motor cognition mechanisms--in particular, overt and covert imitation or mental simulation--are employed in social interaction and can be modeled in artificial embodied agents. Taking a novel perspective on communicative gestures, we develop a hierarchical probabilistic model of motor knowledge that is used to generate behaviors and resonates when seeing them in others. This allows our virtual robot VINCE to learn, perceive and produce meaningful gestures incrementally and robustly in social interaction.
SoCoDiM – A Social Cognition Approach to Dialogue Management
Several higher-level linguistic phenomena like pragmatics (the contribution of context and intention to meaning) or initiative are still mostly unsolved in the field of human-machine dialogue. In the SoCoDiM project we approach these problems from a Social Cognition perspective. We argue that a unified framework for representing and reasoning about the interlocutor's beliefs and intentions is an essential basis for more advanced and believable human-machine dialogue.
Speech-Gesture Alignment
Project B1 of the SFB 673 "Alignment in Communication"
We investigate the cognitive mechanisms that underlie the composition of a multimodal utterance in dialogue. In such utterances, words and gestures are tightly coordinated with respect to their semantics, their form, the manner in which they are performed, their relative temporal arrangement, and their joint organization in a phrasal structure of utterance. We study in particular, both empirically and in computational simulation, how meaning is composed and mapped onto verbal and gestural iconic forms, with a focus on how these processes interact - both within and across these two modalities - to form a coherent multimodal delivery [Video].
VASA – Virtual Assistents and their Social Acceptability
VASA investigates the requirements and capabilities of cognitive interaction technology for assisting people with special needs in everyday situations. A virtual agent offers an intuitive and natural interaction with the technical system; conversation replaces difficult manual handling processes and embodying the system's 'identity', generating a social presence. This increases the acceptance of system-generated suggestions, facilitates a perception of personalized assistance, and enables the perception of continued interaction with the agent as a mutual familiarization process. VASA will ensure the setup of a stable multi-modal interaction system for elderly people for administration of the daily schedule and personalized video communication, in cooperation with the v. Bodelschwingh Foundation Bethel. Initially, personalization capabilities in this interaction paradigm will be investigated, and, subsequently, long-term effects of agent-side initiative and the application of increasingly certain user preferences on the trust in this quotidian companion agent.
Internal Research Platforms
Articulated Communicator Engine (ACE)
ACE is a toolkit for building animated characters that are able to generate realtime multimodal behavior. It provides means of building a character from a kinematic body model, and of specifying behavior by describing its desired overt form in a XML language (MURML, BML). ACE features, e.g., locomotion, reactive behaviors, emotional display, inverse kinematics, keyframing, trajectory formation, and procedural gesture animation. One of its focus is on synthesizing multimodal utterances, for which it takes care of scheduling coverbal gestures, synthetic speech, and facial animations on-the-fly and coordinating them in a natural fashion. ACE consists of a set of C++ libraries, is easily extensible, and independent of any graphics engine. It has been employed in a number of research projects. One of its main application is in the virtual human Max.