The U.S. Patent and Trademark Office officially published a series of 59 newly granted patents for Apple Inc. today. In this particular report we cover a patent relating to eye and head tracking system that could be used in context with playing games, but more importantly, the technology is used for advancing certain aspects of facial analysis relating to Face ID for iDevices, Macs and more.

Apple notes in their granted patent that it has recently been noted that three dimensional (3D) head tracking using a video sequence, or pose estimation using multiple images is an essential prerequisite for robust facial analysis and face recognition.

Eye tracking often forms the basis of these operations and may be thought of as the process of electronically locating the point of a person's gaze, or following and recording the movement of the person's point of gaze. In practice, eye tracking is provided by locating and tracking corneal reflections from an applied light source.

Because infrared or near-infrared light is not perceivable by the human eye, it is often used as the light source; infrared or near-infrared light passes through the pupil but is reflected by the iris, generating a differentiation between the pupil and the iris.

Apple's invention relates to systems, methods to detect and track a user's eye gaze and head movement. In general, techniques are disclosed for identifying a user's pupil location and using this information, in conjunction with a three dimensional (3D) model of the user's head, perform gaze tracking operations.

More particularly, techniques disclosed utilize pupil gradient information to refine an initial pupil location estimate. Once identified, the pupil's location may be combined with 3D head pose information to generate an accurate and robust gaze detection mechanism.

According to Apple, pupil location and gaze tracking operations may be performed by a desktop, laptop, notebook, tablet, smartphone or a gaming device.

A key part of this invention is about Face ID being able to capture a person's facial features and to be able to detect a user's eyes even through glare on glasses which is shown in patent figs. 7A and 7B below.  

Apple's patent FIG. 8 below shows, in block diagram form, a system for performing pupil localization and gaze tracking.

Apple's patent FIGS. 7A and 7B above illustrate pupil glare movement generated by the lighting and camera arrangement.

More specifically, in FIG. 7A, when emitters 1 and 3 are illuminated glare regions #700 and #705 may move in a first direction while, in FIG. 7B, when emitters 2 and 4 are illuminated glare regions #710 and #715 may move in a second (different) direction. By alternatively illuminating the target with emitters 1-3 and 2-4 the glasses-induced glare regions may be caused to move thereby exposing at least one pupil.

Apple's granted patent 10,496,163 was filed in Q3 2017 and published today by the US Patent and Trademark Office. Two of Apple's inventors worked in Apple's Special Projects Group / Computer 3D Vision Group.