• search hit 1 of 1
Back to Result List

Semantic Fusion for Natural Multimodal Interfaces using Concurrent Augmented Transition Networks

Please always quote using this URN: urn:nbn:de:bvb:20-opus-197573
  • Semantic fusion is a central requirement of many multimodal interfaces. Procedural methods like finite-state transducers and augmented transition networks have proven to be beneficial to implement semantic fusion. They are compliant with rapid development cycles that are common for the development of user interfaces, in contrast to machine-learning approaches that require time-costly training and optimization. We identify seven fundamental requirements for the implementation of semantic fusion: Action derivation, continuous feedback,Semantic fusion is a central requirement of many multimodal interfaces. Procedural methods like finite-state transducers and augmented transition networks have proven to be beneficial to implement semantic fusion. They are compliant with rapid development cycles that are common for the development of user interfaces, in contrast to machine-learning approaches that require time-costly training and optimization. We identify seven fundamental requirements for the implementation of semantic fusion: Action derivation, continuous feedback, context-sensitivity, temporal relation support, access to the interaction context, as well as the support of chronologically unsorted and probabilistic input. A subsequent analysis reveals, however, that there is currently no solution for fulfilling the latter two requirements. As the main contribution of this article, we thus present the Concurrent Cursor concept to compensate these shortcomings. In addition, we showcase a reference implementation, the Concurrent Augmented Transition Network (cATN), that validates the concept’s feasibility in a series of proof of concept demonstrations as well as through a comparative benchmark. The cATN fulfills all identified requirements and fills the lack amongst previous solutions. It supports the rapid prototyping of multimodal interfaces by means of five concrete traits: Its declarative nature, the recursiveness of the underlying transition network, the network abstraction constructs of its description language, the utilized semantic queries, and an abstraction layer for lexical information. Our reference implementation was and is used in various student projects, theses, as well as master-level courses. It is openly available and showcases that non-experts can effectively implement multimodal interfaces, even for non-trivial applications in mixed and virtual reality.show moreshow less

Download full text files

Export metadata

Additional Services

Share in Twitter Search Google Scholar Statistics
Metadaten
Author: Chris Zimmerer, Martin Fischbach, Marc Erich Latoschik
URN:urn:nbn:de:bvb:20-opus-197573
Document Type:Journal article
Faculties:Fakultät für Mathematik und Informatik / Institut für Informatik
Language:English
Parent Title (English):Multimodal Technologies and Interaction
ISSN:2414-4088
Year of Completion:2018
Volume:2
Issue:4
Source:Multimodal Technologies Interact. 2018, 2(4), 81; https://doi.org/10.3390/mti2040081
DOI:https://doi.org/10.3390/mti2040081
Dewey Decimal Classification:0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 004 Datenverarbeitung; Informatik
Tag:human-computer interaction; multimodal fusion; multimodal interface; natural interfaces; procedural fusion methods; semantic fusion
Release Date:2020/06/18
Date of first Publication:2018/12/06
Licence (German):License LogoCC BY: Creative-Commons-Lizenz: Namensnennung 4.0 International