Research in the Sociable Agents Group
Our research projects target systems and tools to make machines conversational, cooperative, convergent, and companionable, and to explore these abilities in novel human-machine interaction scenarios.
DiaCoSA – Dialogue Coordination for Sociable Agents
Smooth and trouble-free interaction in dialogue is only possible if interlocutors are able to coordinate their actions with each other. Because of this, we think that an understanding of how human coordination devices work is the basis for successful spoken language computer interfaces such as embodied conversational agents or dialogue systems. In this research project we model two important coordination mechanisms that are commonly found in human-human interaction: feedback and adaptation. Read more …
IMoSA – Imitation Mechanisms and Motor Cognition for Social Embodied Agents
This project explores how the brain's motor cognition mechanisms--in particular, overt and covert imitation or mental simulation of movement--are employed in social interaction and can be modeled in artificial embodied agents. Taking this novel perspective on communicative gestures, we develop a hierarchical probabilistic model of motor knowledge that is used to generate behaviors and resonates when seeing them in others. This allows our virtual robot VINCE to learn, perceive and produce meaningful gestures incrementally and robustly in social interaction.
Speech-Gesture Alignment
Project B1 of the SFB 673 "Alignment in Communication"
This project investigates the cognitive mechanisms that underlie the production of multimodal utterances in dialogue. In such utterances, words and gestures are tightly coordinated with respect to their semantics, their form, the manner in which they are performed, their temporal arrangement, and their joint organization in a phrasal structure of utterance. We study in particular, both empirically and with computational simulation, how multimodal meaning is composed and mapped onto verbal and gestural iconic forms and how these processes interact both within and across both modalities to form a coherent multimodal delivery [Video].
VASA – Virtual Assistents and their Social Acceptability
with Karola Pitsch (Linguistics)
VASA investigates the requirements and capabilities of cognitive interaction technology for assisting people with special needs in everyday situations. In cooperation with the v. Bodelschwingh Foundation Bethel, VASA studies how elderly people as well as people with cognitive disabilities can be supported in managing their daily schedule and personalized video communication. This support is given through a virtual agent that offers an intuitive and natural interaction with the technical system; conversation replaces difficult manual handling processes and embodies the system's 'identity', generating a social presence. We study how this affects the acceptance of system-generated suggestions, facilitates a perception of personalized assistance, and enables the perception of continued interaction as a mutual familiarization process.
ADECO – Adaptive Embodied Communication
with Thomas Schack (Sport Sciences)
Instructions about sequences of actions are better memorized when offered with appropriate gestures. In this project, a virtual character is used to give instructions with self-generated gestures. The instructions are tailored to the specific needs of the user depending on the user's memory representations as assessed using the SDM-A method. The effect of this adaptive embodied action tutoring on the action learning is investigated.
Querschnittsprojekt Mensch-Maschine-Interaktion
Project in the Spitzencluster it's OWL
Ziel des Forschungsprojekts ist die Entwicklung innovativer Methoden und Verfahren für intuitive Bedienschnittstellen von Produkten und Produktionssystemen im Spitzencluster. Dazu werden Methoden und Verfahren der MMI-Spitzenforschung in Form von Lösungsmustern verfügbar gemacht und in einer MMI-Toolbox für die Anwendung in Unternehmen bereitgestellt.
ASAP – Artificial Social Agent Platform
with HMI, University of Twente (Netherlands)
The Articulated Social Agents Platform (ASAP) provides a collection of software modules for social robots and virtual humans jointly developed by the Sociable Agents group (SoA) in Bielefeld and the Human Media Interaction group (HMI) in Twente. In addition to a collection of tools, we also provide the means (through middleware, architecture concepts and shared build and deployment strategies) to compose virtual human or robot applications in which the tools are embedded. Our current work focuses on AsapRealizer, a BML Realizer (behavior generator) for incremental, fluent, multimodal interaction with a virtual human or robot, and IPAACA, a middleware that implements an incremental processing architecture in a distributed fashion.
CLARIN-D – Language and other Modalities
with Hamburg University and RWTH Aachen University
The aim of the project is to make multimodal corpora easily accessible, more transparent and searchable. To this end, three different multimodal corpora are extended and prepared for later integration into a larger research infrastructure: Speech and Gesture Alignment Corpus (SaGA Corpus, Bielefeld University), Dicta-Sign Corpus of German Sign Language (DS-DGS Corpus, Hamburg University) and Natural Media Motion Capture-Data (NM MoCap Corpus, RWTH Aachen University). We work on corpus specific tasks, namely annotations, including intermodal relations, validation and quality control, meta data description, file conversion and documentation, and finally the release of such corpus data. This project is carried out within the F-AG 6 "Language and other Modalities" of the project CLARIN-D “Research Infrastructure for Digital Humanities”.
Completed Projects
Conceptual Motorics
with K. Rohlfing, I. Wachsmuth, F. Joublin; CoR-Lab
A crucial steps in the attempt to build sociable, humanoid robots is to endow them with nonverbal expressivity. This project enabled the humanoid robot ASIMO to flexibly produce synthetic speech along with expressive hand gesture, e.g., to point to objects being referred to or to illustrate actions currently discussed, without being limited to a predefined repertoire of motor actions. This flexibility was exploited in experiments which gave new insights into human perception and understanding of gestural machine behaviors and how to use these in designing artificial communicators. For example, we found that humans attend to a robot's gestures, are affected by incongruent speech-gesture combinations, but socially prefer robots that produce imperfect gesturing.
AMALIS – Adaptive Machine Learning of Interaction Sequences
with Thomas Herrmann (Ambient Intelligence)
Interaction scenarios are full of multivariate sequences of data, e.g., speech, nonverbal behavior streams, data streams from interface operation. This project explored how systems can be enabled to learn such data in highly dynamic environments, such as companion-based interaction with smart rooms, in which unpredictable changes may occur and the systems must re-adjust at many events. We developed models for unsupervised and reinforcement learning of hierarchical structure from sequential data (Ordered Means Models), which not only afford the analysis of sequences but also generation of learned patterns. This was demonstrated, e.g., in enabling the agent VINCE to play, and always win the rock-paper-scissors game.