Project summary

	Project no.: PN-III-P4-ID-PCE-2020-0434; Contract no.: PCE29/2021
	Principal Investigator: Radu-Daniel Vatavu
	Funded by UEFISCDI, Romania
	Funding scheme: PNIII P4 - Fundamental and frontier research, Exploratory research projects (PCE)
	Running period: January 2021 - December 2023 (36 months)

Abstract

The goal of this project is to conceptualize, design, and implement new smart wearables and environments to empower users with new sensorimotor abilities. To this end, we introduce the concept of Sensorimotor Realities (SRs) as a new type of physical-digital reality, distinct from Virtual (VR), Augmented (AR), Mixed (MR), and Mediated Reality (XYR), with grounds in eXtended Reality (XR), motor control theory, and ability-based design. Applications of SRs target people with motor impairments, but also people without impairments, enabling them to experience new sensorimotor abilities with wearable computing, ambient intelligence, and XR immersive technologies.



This figure shows various sensorimotor abilities that enable a diversity of skills: (a) a young adult's precise touch on a small wearable; (b) a six-year-old child assembling tiny Lego pieces; (c) a person with Spinal Cord Injury using a TV remote control by firmly gripping it with both hands; (d) a wheelchair user employing a tablet device by supporting it on his thigh; (e) a blind person moving her hands across a HMD to understand the form factor of the device; and (f) a user immersed in VR manipulating virtual objects. Sensorimotor Realities are a new concept and framework to characterize and foster research contributions and practical systems that mediate skills such as these, while capitalizing on the heterogeneity of human sensorimotor abilities.

Objectives

The main objective of this project is the conceptualization of Sensorimotor Realities, a new type of physical-digital reality that enables augmentation of motor abilities with the conjoint operation of wearable devices, smart environments, and immersive technologies.

Expected results

The main expected result is represented by the concept of Sensorimotor Realities (SRs). Associated results are:

1. Conceptual space of SRs
2. Taxonomy of enhanced sensorimotor abilities
3. Prototypes of wearable devices, designs of smart environments, and interaction techniques for SRs
4. Empirical data from technical evaluations and user studies
5. Scientific publications in high-quality conferences and journals and yearly research reports

Team

Publications

	1.	Adrian-Vasile Catana, Radu-Daniel Vatavu. (2024). Fingerhinter Takes Center Stage: User Experience Insights from Informal Encounters with a Finger-Augmentation Device. Proceedings of AIxVR '24, the 6th IEEE International Conference on Artificial Intelligence & Extended and Virtual Reality. IEEE, USA, 5 pages PDF
	2.	Radu-Daniel Vatavu. (2023). From Natural to Non-Natural Interaction: Embracing Interaction Design Beyond the Accepted Convention of Natural Proceedings of ICMI '23, the 25th ACM International Conference on Multimodal Interaction. ACM, New York, NY, USA, 684-688 PDF | DOI | ACCEPTANCE RATE: 18.8% (3/16) | ARC CORE B | WOS: 001147764700079   1st PLACE BLUE SKY PAPER AWARD
	3.	Alexandre Nevsky, Timothy Neate, Radu-Daniel Vatavu, Elena Simperl. (2023). Accessibility Research in Digital Audiovisual Media: What Has Been Achieved and What Should Be Done Next?. Proceedings of IMX '23, the ACM International Conference on Interactive Media Experiences. ACM, New York, NY, USA, 94-114 PDF | DOI | ACCEPTANCE RATE: 38.9% (14/36) | WOS: 001117075100009   HONORABLE MENTION AWARD
	4.	Radu-Daniel Vatavu. (2023). Leveraging Sensorimotor Realities for Assistive Technology Design Bridging Smart Environments and Virtual Worlds. Proceedings of PETRA '23, the 16th International Conference on Pervasive Technologies Related to Assistive Environments. ACM, New York, NY, USA, 247-253 PDF | DOI | ACCEPTANCE RATE: 51.6% (79/153) | WOS: 001124478300044
	5.	Milad Jamalzadeh, Yosra Rekik, Laurent Grisoni, Radu-Daniel Vatavu, Gualtiero Volpe, Alexandru Dancu. (2023). Effects of Moving Speed and Phone Location on Eyes-Free Gesture Input with Mobile Devices. Proceedings of INTERACT '23, the 19th IFIP International Conference on Human–Computer Interaction. Springer, Cham, 469-478 PDF | DOI | ACCEPTANCE RATE: 31.0% (18/58) | ARC CORE B
	6.	Cristian Pamparau, Ovidiu-Andrei Schipor, Alexandru Dancu, Radu-Daniel Vatavu. (2023). SAPIENS in XR: Operationalizing Interaction-Attention in Extended Reality. Virtual Reality (27). Springer, 1765–1781 PDF | DOI IF: 4.2, 5-Year IF: 5.5, Q1 quartile (JCR 2022) | WOS: 000937114600002
	7.	Adrian-Vasile Catană, Radu-Daniel Vatavu. (2023). Fingerhints: Understanding Users' Perceptions of and Preferences for On-Finger Kinesthetic Notifications. Proceedings of CHI '23, the CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, Article No. 518, 17 pages PDF | DOI | ACCEPTANCE RATE: 27.6% (879/3182) | ARC CORE A* | WOS: 001037809506001
	8.	Laura-Bianca Bilius, Ovidiu-Ciprian Ungurean, Radu-Daniel Vatavu. (2023). Understanding Wheelchair Users' Preferences for On-Body, In-Air, and On-Wheelchair Gestures. Proceedings of CHI '23, the CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, Article No. 78, 16 pages PDF | DOI | ACCEPTANCE RATE: 27.6% (879/3182) | ARC CORE A* | WOS: 001037809504027   HONORABLE MENTION AWARD
	9.	Radu-Daniel Vatavu. (2023). iFAD Gestures: Understanding Users' Gesture Input Performance with Index-Finger Augmentation Devices. Proceedings of CHI '23, the CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, Article No. 576, 17 pages PDF | DOI | ACCEPTANCE RATE: 27.6% (879/3182) | ARC CORE A* | WOS: 001037809504026   HONORABLE MENTION AWARD
	10.	Laura-Bianca Bilius, Ovidiu-Ciprian Ungurean, Radu-Daniel Vatavu. (2023). An Expressivity-Complexity Tradeoff?: User-Defined Gestures from the Wheelchair Space are Mostly Deictic. CHI EA '23, Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, Article No. 35, 8 pages PDF | DOI | ACCEPTANCE RATE: 33.8% (327/967) | ARC CORE A*
	11.	Radu-Daniel Vatavu. (2023). Vulnerabilities of Mediated Embodiment: Towards Unmasking Security and Privacy Risks of Ability-Mediating Wearables. Proceedings of WAMWB '23, the Workshop on Advances of Mobile and Wearable Biometrics, co-located with MobileHCI '23, the 25th ACM International Conference on Mobile Human-Computer Interaction 2023 (Athens, Greece). CEUR-WS vol. 3517, Article no. 6, 65-69 PDF | DOI
	12.	Radu-Daniel Vatavu. (2022). Sensorimotor Realities: Formalizing Ability-Mediating Design for Computer-Mediated Reality Environments. Proceedings of ISMAR '22, the 21st IEEE International Symposium on Mixed and Augmented Reality (Singapore). IEEE, USA, 685-694 PDF | DOI | ACCEPTANCE RATE: 21.1% (93/441) | ARC CORE A* | WOS:000923767800074
	13.	Radu-Daniel Vatavu, Ovidiu-Ciprian Ungurean, Laura-Bianca Bilius. (2022). Interactive Public Displays and Wheelchair Users: Between Direct, Personal and Indirect, Assisted Interaction. Proceedings of UIST '22, the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA). ACM, New York, NY, USA, Article No. 45, 17 pages PDF | DOI | ACCEPTANCE RATE: 26.3% (98/372) | ARC CORE A* | WOS:001046841800051
	14.	Ovidiu-Ciprian Ungurean, Radu-Daniel Vatavu. (2022). Ability-Centered Examination of People with Motor Impairments' Interaction with Television Towards More Accessible Smart Home Entertainment Environments. Proceedings of ISAmI '22, the 13th International Symposium on Ambient Intelligence (L'Aquila, Italy). Lecture Notes in Networks and Systems 603. Springer, Cham, 32-43 PDF | DOI | WOS:000927582100004
	15.	Cristian Pamparau, Radu-Daniel Vatavu. (2022). The User Experience of Journeys in the Realm of Augmented Reality Television. Proceedings of IMX '22, the ACM International Conference on Interactive Media Experiences (Aveiro, JB, Portugal). ACM, New York, NY, USA, 161-174 PDF | DOI | ACCEPTANCE RATE: 40.4% (19/47) | WOS:001117055700015
	16.	Radu-Daniel Vatavu. (2022). Possi(A)bilities: Augmented Reality Experiences of Possible Motor Abilities Enabled by a Video-Projected Virtual Hand. Proceedings of ISEA '22, the 27th International Symposium on Electronic Art (Barcelona, Spain). ISEA International, 825-828 PDF | DOI | ACCEPTANCE RATE: 28.0% (308/1100)
	17.	Bogdan-Florin Gheran, Santiago Villarreal-Narvaez, Radu-Daniel Vatavu, Jean Vanderdonckt. (2022). RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowledge on User-Defined Gestures. Proceedings of AVI '22, the International Conference on Advanced Visual Interfaces (Rome, Italy). ACM, New York, NY, USA, 5:1-5:9 PDF | DOI | ACCEPTANCE RATE: 24.2% (15/62) | ARC CORE B | WOS:001051742000014
	18.	Bogdan Popoveniuc, Radu-Daniel Vatavu. (2022). Transhumanism as a Philosophical and Cultural Framework for Extended Reality Applied to Human Augmentation. Proceedings of AH '22, the 13th Augmented Human International Conference (Winnipeg, MB, Canada). ACM, New York, NY, USA, 6:1-6:8 PDF | DOI | WOS:000944005200003
	19.	Alexandru-Tudor Andrei, Alexandru-Ionut Siean, Radu-Daniel Vatavu. (2022). Tap4Light: Smart Lighting Interactions by Tapping with a Five-Finger Augmentation Device . Proceedings of AH '22, the 13th Augmented Human International Conference (Winnipeg, MB, Canada). ACM, New York, NY, USA, 4:1-4:2 PDF | DOI | WOS:000944005200008
	20.	Radu-Daniel Vatavu. (2022). Are Ambient Intelligence and Augmented Reality Two Sides of the Same Coin? Implications for Human-Computer Interaction. Proceedings of CHI '22 EA, the CHI Conference on Human Factors in Computing Systems Extended Abstracts (New Orleans, LA, USA). ACM, New York, NY, USA, Article no. 362, 8 pages PDF | DOI | ACCEPTANCE RATE: 35.7% (258/722) | ARC CORE A*
	21.	Radu-Daniel Vatavu, Jacob O. Wobbrock. (2022). Clarifying Agreement Calculations and Analysis for End-User Elicitation Studies. ACM Transactions on Computer-Human Interaction 29(1), Article No. 5, pp 1-70 PDF | DOI IF: 3.7, 5-Year IF: 4.6, Q2 quartile (JCR 2022) | WOS:000921867700005
	22.	Radu-Daniel Vatavu, Ovidiu-Andrei Schipor. (2021). Formalizing Digital Proprioception for Devices, Environments, and Users. Proceedings of ISAmI '21, the 12th International Symposium on Ambient Intelligence (Salamanca, Spain). Lecture Notes in Networks and Systems vol. 483. Springer, Cham, 1-10 PDF | DOI
	23.	Alexandru-Ionut Siean, Laura-Bianca Bilius, Radu-Daniel Vatavu. (2021). Assistive Technology in the Synchrony between Ambient Intelligence and Mixed Reality for People with Motor Disabilities. Proceedings of ISAmI '21, the 12th International Symposium on Ambient Intelligence (Salamanca, Spain). Lecture Notes in Networks and Systems vol. 483. Springer, Cham, 23-33 PDF | DOI
	24.	Irina Popovici, Radu-Daniel Vatavu, Pu Feng, Wenjun Wu. (2021). AR-TV and AR-Diànshì: Cultural Differences in Users’ Preferences for Augmented Reality Television. Proceedings of IMX '21, the ACM International Conference on Interactive Media Experiences (Virtual Event, NY, USA). ACM, New York, NY, USA, 50-60 PDF | DOI | ACCEPTANCE RATE: 42.5% (17/40) | WOS: 001117065300006   BEST PAPER AWARD
	25.	Ovidiu-Ciprian Ungurean, Radu-Daniel Vatavu. (2021). Coping, Hacking, and DIY: Reframing the Accessibility of Interactions with Television for People with Motor Impairments. Proceedings of IMX '21, the ACM International Conference on Interactive Media Experiences (Virtual Event, NY, USA). ACM, New York, NY, USA, 37-49 PDF | DOI | ACCEPTANCE RATE: 42.5% (17/40) | WOS: 001117065300005
	26.	Radu-Daniel Vatavu. (2021). Accessibility of Interactive Television and Media Experiences: Users with Disabilities Have Been Little Voiced at IMX and TVX. Proceedings of IMX '21, the ACM International Conference on Interactive Media Experiences (Virtual Event, NY, USA). ACM, New York, NY, USA, 218-222 PDF | DOI | WOS: 001117065300024

Other publications: Workshops, Invited papers, Panels

	27.	Adrian-Vasile Catana, Cristian Pamparau, Radu-Daniel Vatavu. (2022). Using Mediated Reality to Induce Sensorimotor Impairments. ASSETS '22 Workshop on Multidisciplinary Perspectives on Designing Accessible Systems for Users with Multiple Impairments: Grand Challenges and Opportunities for Future Research (Athens, Greece/virtual). PDF
	28.	Radu-Daniel Vatavu. (2022). Designing Interactive Computer Systems within the Framework of Sensorimotor Realities. Invited Paper at ECCO '22, the International Conference on Electronics, Communications, and Computing (Hybrid Event, Chisinau, Republic of Moldova). PDF | DOI
	29.	Radu-Daniel Vatavu. (2022). Designing Interactive Experiences in the Interplay between Ambient Intelligence and Mixed Reality Proceedings of CHI '22 Extended Abstracts, the CHI Conference on Human Factors in Computing Systems Extended Abstracts (New Orleans, LA, USA). ACM, New York, NY, USA, Article no. 140, 1-3 PDF | DOI | ARC CORE A*
	30.	David Geerts, Radu-Daniel Vatavu, Alisa Burova, Vinoba Vinayagamoorthy, Martez Mott, Michael Crabb, Kathrin Gerling. (2021). Challenges in Designing Inclusive Immersive Technologies Proceedings of MUM '21, the 20th International Conference on Mobile and Ubiquitous Multimedia (Leuven, Belgium). ACM, New York, NY, USA, 182-185 PDF | DOI | WOS: 001112164200021 | ARC CORE B

Awards

Media

Project reports (in Romanian)

1. Final scientific report (2021-2023) PDF
2. Scientific report 2022 PDF
3. Scientific report 2021 PDF

Other resources (software, diagrams, and datasets)

1. Mind map for the concept of Sensorimotor Realities: PDF
   
   Artwork copyright © 2021 Radu-Daniel Vatavu.
2. Six-dimensional conceptual space for Sensorimotor Realities: v1 PDF | v2 PDF
   
   Artwork copyright © 2021-2022 Radu-Daniel Vatavu.
3. Venn diagram positioning Sensorimotor Realities with respect to other reality concepts from the XR landscape: PDF
   Artwork copyright © 2021 Radu-Daniel Vatavu.
4. The iFAD gesture taxonomy for index-finger augmentation devices. PDF
   Artwork copyright © 2023 Radu-Daniel Vatavu.
5. Source code for the HoloLens ARTV application (ISMAR '22): GitLab
   Developed by and copyright (2022) Cristian Pamparau.
6. R software that implements statistical tests for analyzing data resorting from end-user elicitation studies. More details are available in the ACM Transactions on Computer-Human Interaction paper. The software is distributed under the New BSD license agreement.
   
7. SAPIENS-in-XR, a software architecture to support design and engineering of interactive systems that implement peripheral input in eXtended Reality (XR) environments. We introduce our online web-based software application for peripheral interactions in XR, which enables practitioners to have access to JavaScript code and observe how JSON messages are being exchanged by the various software components of the architecture. More details are available on the dedicated web page of SAPIENS-in-XR and in the corresponding paper in the Virtual Reality journal.
   The software resources are released under this license .
8. The Fingerhints dataset & R software consisting of 210 fingerhints elicited from 21 participants to effect various notification types, e.g., text message, social media, or news delivery. More details are available in the 2023 Fingerhints paper published at CHI 2023 .
   
   The dataset and corresponding software are released under this license .
9. The iFAD dataset & C# software consisting of 6,639 gestures collected from 20 participants using an index-finger augmentation device. More details are available in the 2023 iFAD paper published at CHI 2023 .
   
   The dataset and corresponding software are released under this license .
10. The on-body, on-wheelchair dataset & R software consisting of 231 gestures collected from 11 wheelchair users to effect 21 referents representing common actions, types of digital content, and navigation commands for interactive systems. More details are available in the 2023 on-body, on-wheelchair gestures paper published at CHI 2023 .
    
    The dataset and corresponding software are released under this license .