Apple invents Eye-Gaze based Biofeedback to determine and correct a user's attentive state to enhance entertainment, education & more
On Tuesday, Patently Apple posted a report titled "Apple is reportedly developing an AI-Driven Mood Tracking & Health Coach App and/or Service." In that report we pointed to Bloomberg's Mark Gurman stating that "Apple Inc. is working on an artificial intelligence-powered health coaching service and new technology for tracking emotions, its latest attempt to lock in users with health and wellness features." More specifically, "The new coaching service — codenamed Quartz — is designed to keep users motivated to exercise, improve eating habits and help them sleep better." Along this same line of thinking, a new Apple patent in Europe expands on their wellness focus in another direction.
Earlier this morning, Patently Apple discovered a patent application from Apple titled "Eye-Gaze based Biofeedback," that was published in Europe on April 12, 2023. The patent relates to displaying content with an XR Headset to determine a user's attentive state during the display of content.
Apple's patent background states that a user's cognitive or attentive state while viewing content on an electronic device, can have a significant effect on the user's experience.
For example, staying focused and engaged while using a head-mounted device (HMD) in an extended reality (XR) environment may be required for meaningful experiences, such as meditation, learning a new skill, watching educational or entertaining content, or reading a document. Improved techniques for assessing the attentive states of users viewing and interacting with content may enhance a user's enjoyment, comprehension, and learning of the content. Moreover, content may not be presented in a way that makes sense to a particular user. Content creators and display systems may be able to provide better and more tailored user experiences that a user is more likely to enjoy, comprehend, and learn from based on being able to determine the user's cognitive state.
Apple's patent covers devices, systems, and methods that determine an attentive state of a user during or after the presenting of content (e.g., visual and/or auditory content that could include real-world physical environment, virtual content, or a combination of each) based on the user's gaze characteristic(s) to enhance the experience.
While the patent may focus on the use of an XR Headset, Apple notes that their system could also apply to an iDevice, MacBook and iMac/desktop in providing content (e.g., a visual and/or auditory experience) to the user and obtains, with a sensor, physiological data (e.g., gaze characteristics) associated with a response of the user to the content.
Based on the obtained physiological data, the techniques described can determine a user's attentive state (e.g., meditating vs. mind wandering) during or after the presenting (e.g., a meditation experience) by classifying the user's gaze characteristic(s).
Based on the classified gaze characteristic(s), the techniques can provide feedback to the user that the current attentive state differs from an intended attentive state of the content, recommend similar content or similar portions of the content, and/or adjust content or feedback mechanisms corresponding to the content based on the current attentive state differing from an intended attentive state of the content.
In an exemplary implementation, the integration of meditation and mindfulness practices with techniques described in this patent can enhance a meditation experience by providing individuals with real-time feedback on meditation performance.
Staying focused and engaged during meditation while using an HMD in an extended reality (XR) setting can improve the user's meditation practice and help a user gain the benefits associated with meditation.
For example, novices who are interested in meditating may have trouble staying on task during a meditation session, and they may benefit from accurate feedback on their performance. The techniques described in this patent can detect when users get distracted while meditating based on a user's variability in gaze characteristics.
Finding defined markers of attention lapses and providing performance feedback during meditation could enhance a user experience, provide additional benefits from the meditation session, and provide a guided and supportive teaching approach (e.g., a scaffolding teaching method) for users to advance through their meditation practice.
Eye gaze patterns can vary based on the attention state of an individual and characteristics of the scene in front of them. These patterns can be obtained while using an HMD with eye tracking technology while users perform mindfulness tasks that demand varying levels of attention, such as focused attention to breath meditation.
Over the course of a meditation session, attention can fluctuate towards and away from the breath. When an individual focuses on their breath during an eyes-open meditation task, eye gaze may be slow in time and constrained to a particular area of the scene. When the individual loses focus and starts to mind wander, eye gaze stability may decrease and/or eye gaze velocity may increase.
Observing repeated measures of eye gaze velocity and position over a duration of time (seconds to minutes) can give insights about the underlying attention state of the user at different time scales. These metrics of attention can be used to provide feedback during a meditation experience.
Apple's patent FIG. 1 illustrates a device displaying a visual experience and obtaining physiological data from a user; Figure 2 illustrates a pupil of the user of Figure 1 in which gaze characteristic(s) varies with time. Figure 3 is a flowchart representation of a method for classifying variability of gaze characteristic(s) to determine an attentive state of a user.
Apple's patent FIGS. 4A and 4B above are charts illustrating variability of a gaze characteristic of a user during two different attentive states; Figure SA and 5B are charts illustrating variability of a gaze characteristic of a user during two different attentive states.
In Apple's patent Figure 6 below we're able to view a block diagram illustrating device components of an exemplary device; Figure 7 is a block diagram of an example head-mounted device (HMD)
Apple's patent that was filed for in 2020 was published in Europe on April 12 2023 under number EP4162345.
Developers, content creators and educators may find Apple's invention of great interest knowing that an advanced Eye-Gaze based Biofeedback system for an HMD could benefit their future customers and/or students remain in a deeper attentive state while viewing content and enhance their learning and/or entertainment experience.
Comments