Apple Invention Describes Face ID Camera System used for Biometrics, Gaze Tracking & Hand Gesturing on Macs or TV
Today the US Patent & Trademark Office published a patent application from Apple that relates to input-output circuitry of a television or computer display that may include a gaze tracking system that gathers point-of-gaze information, vergence information, and head position information. This system may also include a biometric sensor, capture hand gestures and more.
Apple's invention covers an electronic device with a display that uses input-output circuitry to gather input from a viewer. The input-output circuitry may include a gaze tracking system that gathers point-of-gaze information, vergence information, and head position information, may be a biometric sensor, may be an input device such as a button or touch sensor, may capture hand gestures, and/or may gather other information. This information may be used by control circuitry in the electronic device to dynamically adjust the display.
The display may include a pixel array for producing images. An adjustable reflectance and transmittance layer may overlap the pixel array. The adjustable reflectance and transmittance layer may have a linear polarizer, reflective polarizers, an adjustable liquid crystal layer for controlling polarization rotation, and a switchable polarizer. The switchable polarizer may include liquid crystal molecules and dichroic dye molecules.
Control circuitry in the electronic device may individually adjust different areas of the adjustable reflectance and transmittance layer by supplying control signals to the adjustable liquid crystal layer and to the switchable polarizer in each of these areas. The control circuitry may place each area in a reflective mirror mode or in a content-displaying mode. The locations of mirror mode regions and content-displaying regions may be moved with respect to each other in response to information from the input-output circuitry.
Apple's patent FIG. 11 below is a perspective view of an illustrative display with a first area that is displaying an icon or other content and a second area that is being operated in a reflective mode while a viewer's point-of-gaze is being measured using a gaze detection system.
More specifically, the user / viewer #44 is looking at the display in two different directions. Initially, the viewer's point-of-gaze is directed towards mirror region #220. When the viewer's gaze is oriented in this way, a reflection of the viewer will be visible to the viewer. If the viewer desires to view content, the viewer may shift the point-of-gaze towards an icon or other information in content-displaying region #222.
The control circuitry can then use the information from the tracking system to detect when the viewer's point-of-gaze is directed towards region #222 (or other appropriate area of the display) for more than a predetermined amount of time and can conclude that the viewer is commanding the device to convert some or all of mirror region M into a content-displaying region.
Apple's patent FIG. 13 is a flow chart of illustrative steps involved in operating a device of this type.
More specifically, at step 3280, the device may gather information from the viewer. As an example, the input-output circuitry (e.g., tracking system, sensors and/or other input-output devices etc.) may gather head position information, point-of-gaze information (point-of-gaze location, point-of-gaze dwell time and movement information, etc.), gestures (e.g., hand gestures), voice command input, biometric information (facial recognition information), vergence information, health information (directly measured and/or relayed heart rate measurement, respiration rate information, eye movement statistics, etc.), and/or other information on the viewer.
While Apple's patent figure focused on a Mac or TV display, they make it clear that their invention can also apply to an iPhone, iPad or Apple Watch.
The control circuitry may take suitable actions based on this information at step #282 of FIG. 13 above. For example, the control circuitry may use information on the position of the head of the viewer to reconfigure the mirror and content-displaying portions of the display, may use point-of-gaze information, gestures, voice commands, biometric and/or health information, button press information, vergence information, and/or other viewer input and/or environmental data to adjust which content is displayed in the content-displaying portions of the display to add or remove calendar entries, to adjust settings in a messaging application, to set or clear reminders, to capture self-portraits and/or other images, and/or to perform other tasks.
Apple's patent application was originally filed back in Q4 2017. Considering that this is a patent application, the timing of such a product to market is unknown at this time.
Apple has a number of patents covering eye tracking and hand gesturing used with computers and/or television on record with a few examples below:
01: Apple Wins Patent for Controlling a Display or Television by Mere Pointing Gestures
02: Apple Introduces a 3D Component to Assist Head and Eye Tracking in Future Desktops
03: Apple Wins a Patent for a 3D Camera System for Controlling Games, Apps and Possibly Facial Recognition
It was reported a year ago that Apple may have acquired a company specializing in Gaze / Eye Tracking for future mixed reality glasses. So it's clear that Apple is looking at this technology holistically across their devices and beyond.
Patently Apple presents a detailed summary of patent applications and/or granted patents with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.