Today the US Patent & Trademark Office published another series of  patent applications relating to present and/or future iterations of Apple Vision Pro's eye-tracking system, lenticular displays, hand-tracking and more. The main patent application covered in this report covers devices, systems, and methods that determines an eye characteristic (e.g., gaze direction, eye orientation, etc.) based on the determined location of the glint. Apple's Eye-Tracking technology is at the very core of what makes Apple Vision Pro so magical.

In Apple's patent background they note that existing eye-tracking techniques analyze glints that are reflected off of a user's eye and captured via an image sensor. Glints depicted in images captured by such image sensors may be saturated, for example, in circumstances in which relatively strong glint-producing lights are reflected from relatively specular eye surfaces. Such glint saturation may prevent existing eye-tracking techniques from accurately determining glint locations since the saturation may inhibit the ability to measure the center of mass of a glint (e.g., glint centroid) accurately.

Such eye-tracking systems may lack accuracy and be sensitive to mechanical shocks. Thus, it may be desirable to provide a means of efficiently providing more accurate identification of the glints, e.g., by more accurately finding glint centroids.

Glint Analysis using Multi-Zone Lens

Apple's invention covers devices, systems, and methods that determine an eye characteristic (e.g., gaze direction, eye orientation, etc.) based on a determination of a location of a glint. An eye characteristic may be tracked or otherwise determined using glints reflected from an eye and captured by an image sensor. In some aspects, the glint tracking system uses a multi-zone lens that has a zone that spreads out a glint's energy received at the image sensor. Spreading the energy may improve eye tracking by enabling more accurate identification of a glint centroid using light spread from otherwise saturated field points (e.g., light spread into a halo in a point spread function). Additionally, spreading the energy using a multi-zone lens may further reduce errors resulting from lens aberrations, source power, aperture decenter, and the like.

In general, one innovative aspect of this invention can be embodied in methods that include the actions of, at an electronic device having a processor, producing a glint by producing light that reflects off a portion of an eye, receiving the reflected light at a sensor, wherein the reflected light is received after passing through a multi-zone lens having a first zone and a second zone, the first zone and second zone having different energy-spreading characteristics, determining a location of the glint based on the reflected light received at the sensor, and determining an eye characteristic based on the determined location of the glint.

In some aspects, the first zone has a radius of curvature that differs from a radius of curvature of the second zone. In some aspects, the radius of curvature of the first zone spreads the received light more than the radius of curvature of the second zone in a point spread function (PSF). In some aspects, the radius of curvature of the first zone produces a distributed energy portion in the PSF by distributing energy of otherwise saturated field points. In some aspects, the distributed energy portion is a halo.

In some aspects, the first zone produces a distributed energy portion in a point spread function (PSF), and the second zone produces a focused energy portion in the PSF. In some aspects, the first zone includes a tilt such that the distributed energy portion is separate from a focused energy portion in the PSF.

In some aspects, determining the location of the glint includes determining a centroid of the received light. In some aspects, determining the eye characteristic includes determining locations of multiple portions of the eye based on determining locations of multiple glints.

In some aspects, the light is infrared (IR) light. In some aspects, the sensor includes an image sensor and receiving the reflected light includes receiving the reflected light from image data from the sensor. In some aspects, the electronic device is a head-mounted device (HMD).

Apple's patent FIGS. 3 below illustrates the effect of saturation upon determination of a glint location; FIG. 4 illustrates how use of a lens in accordance with some implementations enables more accurate determination of a glint location of an otherwise saturated glint.

Apple's patent FIGS. 8A-8C above illustrate example multi-zone lens for an eye-tracking system; FIG. 10 illustrates an example coma aberration with a multi-zone lens for an eye-tracking system.

To review the patent's full details, review patent application 20230324587.

A Few Other HMD Related Patents Published Today

20230326380 - Boundary Smoothing In A Display: The patent covers a lenticular display that may have a lenticular lens film formed over an array of pixels. The lenticular lenses may be configured to enable stereoscopic viewing of the display such that a viewer perceives three-dimensional images.

20230324690 - Systems With Supplemental Illumination: A head-mounted device may have a head-mounted support structure. Displays may present images to eye boxes at the rear of the head-mounted support structure. Cameras and other sensors may be supported by the head-mounted support structure. Tracking cameras may be used to track the movement of a user's hands or other external objects.

20230324988 - Display Calibration: The patent covers devices, systems, and methods that assess calibration between eye tracking and other components of a head-mounted device (HMD) using another device, such as an iPhone.