Sapiens is a software architecture built on top of Euphoria to support engineering of interactive systems for peripheral input in the context of interacting with smart environments. Sapiens features dedicated components for user and device tracking, attention detection, priority management for devices and notifications, context awareness inference, user interruptibility prediction, and device interchangeability that can be instantiated at will according to the needs of the application implementing peripheral interaction.

Scenario

It is Friday evening. Sandra arrives home and wishes to relax after a hard day at work. She puts her smartphone on the coffee table, lays on the couch, and instructs the smart environment to play her favorite music using a voice command, e.g., "Ambient, play music." The notes of her preferred composer fill the room. After a few minutes, Sandra pulls out her tablet and resumes reading a book from where she left it last time. While she becomes intrigued by the action from the book, she feels that the ambient music is distracting and decides to pause it, "Ambient, stop music," while continuing to read. During this time, various notifications arrive on her smartphone, but because the smart environment knows that Sandra is involved in another activity and that those messages are low priority, they are subtly transferred to a nearby display, situated at the periphery of Sandra’s attention, instead of the smartphone interrupting her reading. However, when a text message from her daughter arrives, it is immediately delivered by the surround sound system, while also shown directly on the tablet, where Sandra is reading her book. Sandra leaves the house to pick up her daughter who has finished her karate lessons a bit earlier today.

Architecture design

Besides the designing requirements borrowed from Euphoria, we propose the following four attention-related properties for the Sapiens architecture that are specific to peripheral interaction and not implemented by default under Euphoria:

1. Multimodal orientation.Processing of events and decision making in Sapiens should reflect the multimodal nature of human attention and, consequently, offer an appropriate conceptual framework for peripheral interaction that addresses not just one single sense, but multiple senses at once towards an effective user experience. For example, when an important, highpriority notification arrives on Sandra’s smartphone, the message is delivered using visual and auditory feedback via the smartphone, surround sound system, and the tablet on which Sandra is reading the e-book. Other, low-priority notifications are delivered in ways that do not interrupt Sandra’s current reading, such as a gentle rendering of the notifications on the wall display, accessible to Sandra’s periphery of attention.
2. Priority inference represents the ability of Sapiens to distinguish various demands for the user’s attention coming from various devices and prioritize what information to deliver users. For example, low-priority notifications should not be disturbing to Sandra’s reading, while important messages should be addressed to Sandra’s focus of attention using devices that already capture or are in the immediate vicinity of Sandra’s attentional field, such as the tablet or the surround sound system.
3. Probabilistic response. Human attention can be distributed across several tasks, each having a specific cognitive load. For example, while Sandra is reading on the tablet and listening to music delivered via the surround sound system, the attention is obviously distributed. However, it is reasonable to assume that reading demands more cognitive resources and, therefore, is the primary task that captures Sandra’s focus of attention. Therefore, a probabilistic characterization of the attentional processes is thus recommendable.
4. Proactivity underlines the capacity of Sapiens not just to infer the user’s focus of attention, but also to predict changes. This design requirement is important for an application built on top of Sapiens to know when to switch between the main task and other secondary ones. Our suggestion to implement this requirement is through the formalism of “hazard functions,” which describe the probability of an event to terminate in the immediate future.

Publications

1. Ovidiu-Andrei Schipor, Radu-Daniel Vatavu, Wenjun Wu. (2019). SAPIENS: Towards Software Architecture to Support Peripheral Interaction in Smart Environments. Proceedings of the ACM on Human-Computer Interaction 3, EICS. ACM, New York, NY, USA, Article No. 11, 24 pages
2. Ovidiu-Andrei Schipor, Radu-Daniel Vatavu, Wenjun Wu. (2019). Integrating Peripheral Interaction into Augmented Reality Applications. In Proceedings of ISMAR '19 Adjunct, the 18th International Symposium on Mixed and Augmented Reality. IEEE Press, 341-342

Demo and source code

We introduce our online simulation application for the Sandra scenario, which enables practitioners to have access to JavaScript code and observe how JSON messages are being exchanged by the various components of Sapiens.