Today the US Patent & Trademark Office published a patent application from Apple that describes systems and methods for monitoring and detecting head gestures performed by a user wearing smartglasses or a head mounted display device that is equipped with multiple light-sensing assemblies. Today, the Apple Watch is pushing into the health arena, especially regarding the monitoring of a user's heart. In the future, an Apple Watch or iPhone, will be able to work with Apple smartglasses to provide biomedical data that is simply mind boggling in scope.   

The vast majority of the patent is focused on the sensor system that will provide biomedical data to the user. At one point it even describes monitoring jaw movements.

Apple notes that sensor data from the photodiodes can capture expansion, contraction, and/or any other suitable movement in the tissue of the user during a head gesture. Various different head gestures, including, but not limited to, chewing, blinking, winking, smiling, eyebrow raising, jaw motioning (e.g., jaw protrusion, jaw retrusion, lateral jaw excursion, jaw depression, jaw elevation, etc.), mouth opening, and/or the like, may be detected by recognizing patterns in sensor data that may be characteristic of each head gesture, as head tissue may expand and contract and anatomical features in the tissue may move during such user gestures."

And then out of nowhere, the engineers will describe a scenario where the device monitoring your heart at a gaming event and once it registers that your heart beats are faster because you're excited some action taking place, the camera in the glasses will automatically begin taking photos knowing something meaningful is occurring.  But this is a rarity, as the patent dives into tracking a user's biometrics to a ridiculous level measuring things like your temples and skull skin.

While the patent focuses on smartglasses and/or a Head Mounted Display, Apple notes that the device could also cover future devices such as reading glasses, sunglasses, athletic goggles, welding goggles, helmet or the like.

These devices may provide one or more light sensing assemblies operative to detect light reflected by and/or transmitted through a portion of the user's head using photoplethysmography or "PPG" now used on Apple Watches.

For example, the head-wearable electronic device may include any suitable number of photodiodes or any other suitable type(s) of optical sensors, each of which may be provided at a respective different location along the structure of the head-wearable device in order to sense light at a respective different position along the head of the user when the device is worn by the user's head.

Due to such positioning, the sensor data from the light sensors can capture movement of anatomical features in the tissue of the head of the user and can be used to determine any suitable gestures of the user, where such user gestures or head gestures may refer to any suitable gestures (e.g., voluntary and/or involuntary gestures of any suitable portion(s) of a user's head) and/or motions and/or actions and/or vocalizations and/or emotions and/or thoughts and/or brain functions and/or heart rate characteristics and/or other biometric characteristics of the user.

Further, different light emitters of the device can emit light at different wavelengths (e.g., infrared ("IR") light, green light, etc.), which may penetrate to different depths in the tissue of the user's head before reflecting back to or otherwise being detected by the photodiodes of the device.

As another example, a method for monitoring a user wearing a head-wearable electronic device (HWD) on the user's head using a head gesture model custodian system is provided that may include initially configuring, at the head gesture model custodian system, a learning engine, receiving, at the head gesture model custodian system from the HWD, HWD sensor category data for at least one HWD sensor category for a head gesture and a score for the head gesture, training, at the head gesture model custodian system, the learning engine using the received HWD sensor category data and the received score, accessing, at the head gesture model custodian system, HWD sensor category data for the at least one HWD sensor category for another head gesture, scoring the other head gesture, using the learning engine for the HWD at the head gesture model custodian system, with the accessed HWD sensor category data for the other head gesture, and, when the score for the other head gesture satisfies a condition, generating, with the head gesture model custodian system, control data associated with the satisfied condition.

Apple's patent FIG. 1 below is a schematic view of an illustrative system 1 that includes a head-wearable electronic device 100 for monitoring a user.

The Sensor Assembly

The sensor assembly is a key part of this invention. Apple notes that the sensor assembly #114 may include any suitable sensor components or subassemblies for detecting any suitable biometric data and/or health data and/or sleep data and/or mindfulness data and/or the like of a user of the Head Wearable Device (HWD 100).

For example, the sensor assembly may include any suitable biometric sensor that may include, but is not limited to, one or more health-related optical sensors, capacitive sensors, thermal sensors, electric field ("eField") sensors, and/or ultrasound sensors, such as photoplethysmogram ("PPG") sensors, electrocardiography ("ECG") sensors, galvanic skin response ("GSR") sensors, posture sensors, stress sensors, photoplethysmogram sensors, and/or the like.

These sensors can generate data providing health-related information associated with the user. For example, PPG sensors can provide information regarding a user's respiratory rate, blood pressure, heart rate ("HR"), heart rate variability ("HRV"), and/or oxygen saturation. ECG sensors can provide information regarding a user's heartbeats. GSR sensors can provide information regarding a user's skin moisture, which may be indicative of sweating and can prioritize a thermostat application to determine a user's body temperature. In some examples, each sensor can be a separate device, while, in other examples, any combination of two or more of the sensors can be included within a single device.

For example, a gyroscope, accelerometer, photoplethysmogram, galvanic skin response sensor, and temperature sensor can be included within a wearable electronic device, such as an HWD, while a scale, blood pressure cuff, blood glucose monitor, SpO2 sensor, respiration sensor, posture sensor, stress sensor, and asthma inhaler can each be separate devices.

While specific examples are provided, it should be appreciated that other sensors can be used and other combinations of sensors can be combined into a single device. Using one or more of these sensors, the HWD can determine physiological characteristics of the user while performing a detected activity, such as a heart rate of a user associated with the detected activity, average body temperature of a user detected during the detected activity, any normal or abnormal physical conditions associated with the detected activity, or the like. In some examples, a GPS sensor or any other suitable location detection component(s) of the HWD can be used to determine a user's location (e.g., geo-location and/or address and/or location type (e.g., library, school, office, zoo, etc.) and movement, as well as a displacement of the user's motion.

An accelerometer, directional sensor, and/or gyroscope can further generate activity data that can be used to determine whether a user of the HWD is engaging in an activity, is inactive, or is performing a gesture.

Any suitable activity of a user may be tracked by the sensor assembly including, but not limited to, steps taken, flights of stairs climbed, calories burned, distance walked, distance run, minutes of exercise performed and exercise quality, time of sleep and sleep quality, nutritional intake (e.g., foods ingested and their nutritional value), mindfulness activities and quantity and quality thereof (e.g., reading efficiency, data retention efficiency), any suitable work accomplishments of any suitable type (e.g., as may be sensed or logged by user input information indicative of such accomplishments), and/or the like. HWD 100 can further include a timer that can be used, for example, to add time dimensions to various attributes of the detected physical activity, such as a duration of a user's physical activity or inactivity, time(s) of a day when the activity is detected or not detected, and/or the like.

At the end of the day Apple has many patents on record describing the future use of smartglasses and various Head Mounted Displays for many kinds of possible future applications. This is the first invention focusing on biometric smartglasses

Apple's patent application that was published today by the U.S. Patent Office was originally filed back in Q3 2018.  Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Inventors

James Stark: Biomedical Engineer

Nicholas Allec: Sensing Engineer

Joel Armstrong-Munger: Produce Design Engineer

About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.