The U.S. Patent and Trademark Office officially published a series of 52 newly granted patents for Apple Inc. today. In this particular report we cover a single granted patent relating to a gaze tracking system for a mixed reality (MR) headset and a host of other possible devices.

Apple notes that devices such as head-mounted displays and computers may have gaze tracking systems that monitor the gaze of a user's eyes. Information on a user's gaze may be used as input for controlling the operation of a head-mounted display or other device.

Due to movement of a head-mounted display relative to a user's eyes or other changes in the operating environment of an electronic device with gaze detection, gaze detection accuracy may be degraded. If care is not taken, this can lead to difficulties in controlling the operation of the electronic device using gaze information.

Apple's invention covers electronic devices that may be provided with displays and gaze tracking systems. The gaze tracking systems is designed to gather information on a user's eyes such as information on the location of the centers of a user's pupils, information on corneal reflection locations and other reflections, and other eye position information associated with the direction in which the user is currently gazing. The direction in which the user is currently gazing can be used in determining the location on the display at which the user is focused (the user's on-screen point of gaze). The user's point of gaze on the display can be used as an input such as an input in a video game, an input in a media creation application, an input for a web browser or business application, or an input in any other suitable type of software running on the electronic device.

Electronic devices that include displays and gaze tracking systems may include, for example, head-mounted displays (e.g., head-mounted devices such as virtual reality or augmented reality glasses), cellular telephones, tablet computers, head-up displays in vehicles and other environments, laptop computers, desktop computers, televisions, wristwatches, and other suitable electronic equipment.

In devices such as head-mounted displays, the general location of a user's eyes will be relatively constant during operation, facilitating accurate eye tracking. In general, however, any suitable electronic device may be provided with a gaze tracking (eye tracking) system (e.g., a stand-alone computer, a head-up display, a portable device, etc.).

Control circuitry in the electronic device can produce a saliency map for content being displayed on the display. The saliency map may be used to identify items of visual interest within the displayed content. The saliency map may identify items such as selectable buttons, text, and other items of visual interest. User input such as mouse clicks, voice commands, and other commands may be used in conjunction with gaze detection system output in determining which objects are being viewed by a user. This allows the control circuitry to accurately determine the location of the user's actual on-screen point of gaze.

Information on the on-screen point of gaze that is determined in this way can be compared to a point of gaze location associated with the measured eye position information from the gaze tracking system. This allows the control circuitry to perform real time calibration operations on the gaze tracking system.

Apple's patent FIG. 2 below is a top view of an illustrative head-mounted display;  FIG. 3 is a perspective view of an illustrative display that is being used to display content for a user while the user's point of gaze on the display is being monitored using a gaze tracking system in accordance with an embodiment.

Apple's patent FIG. 5 below is a flow chart of illustrative operations involved in calibrating a gaze tracking system during normal operation of an electronic device.

Apple's granted patent was originally filed in Q4 2017 and published today by the US Patent and Trademark Office.

Other recent gaze tracking patents include:

01: Apple Advances Gaze Controls as a Key Input Function along with Touch & In-Air Gesturing to Control a Headset

02: Apple Wins a Key Patent Relating to Eye Tracking in Smartglasses

03: "Apple's Fourth Major Mixed Reality Headset Patent Uncovers new Layers of Deep Gaze & Retina Technologies." Today's cover graphic came from this patent.

04: Apple Invention Describes Face ID Camera System used for Biometrics, Gaze Tracking & Hand Gesturing on Macs or TV

For other patents on this subject matter could be found in our archives titled "HMDs, Smartglasses +."

A few of the Inventors

Nicolas Bonnier: Display, Color and Image Processing Engineer. Bonnier was previously employed at Canon.  

Hesam Najafi: Senior Computer Vision Engineer. Najafi was previously employed at Qualcomm

About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.