Earlier today Patently Apple discovered a patent application from Apple from the German Patent Office that was published on January 05, 2023. The patent application relates to gaze tracking technology that could be used in Apple's future mixed reality headset, projection systems in a vehicle's HUD system or even with contact lenses. More specifically, Apple's patent relates to systems, methods, and computer readable media to refine and smooth eye tracking data for improved performance in eye tracking techniques. Four out of the six inventors came to Apple in 2015 when Germany's Metaio was acquired. Metaio was one of the world's pioneers in Augmented Reality and Hologram technologies.  

​Apple notes that in one or more embodiments, a pupil position is maintained within a predetermined range. A lens through which a pupil is monitored may have a predetermined range, which is a known calibrated range, while other regions of the lens may be less calibrated. As such, the pupil location within the calibrated range should be maintained.

​In one or more embodiments, an initial pupil location is determined with respect to a lens located in front of the eye. A change in the pupil position can be detected with respect to the lens at an updated pupil position with respect to the lens. If the updated pupil location is outside the determined bounding box, for example assigned to a calibrated region, a replacement pupil location is selected within the bounding box. ​The updated pupil location is then used for eye tracking techniques.

In some embodiments, the eye tracking system to avoid jitter may change the pupil location over a series of frames such that the pupil location transitions from the original pupil location to the replacement pupil location, for example, based on a time-based sensing function.

​In one or more embodiments, a pupil position is maintained within a predetermined range. A lens through which a pupil is monitored may have a predetermined range, which is a known calibrated range, while other regions of the lens may be less calibrated. As such, the pupil location within the calibrated range should be maintained.

​In one or more embodiments, an initial pupil location is determined with respect to a lens located in front of the eye. A change in the pupil position can be detected with respect to the lens at an updated pupil position with respect to the lens. If the updated pupil location is outside the determined bounding box, for example assigned to a calibrated region, a replacement pupil location is selected within the bounding box.

​The updated pupil location is then used for eye tracking techniques. In some embodiments, the eye tracking system to avoid jitter may change the pupil location over a series of frames such that the pupil location transitions from the original pupil location to the replacement pupil location, for example, based on a time-based sensing function.

​Apple's patent FIGS. 1A-B below show example diagrams of a structure for eye tracking and performing pupil localization.

Further into Apple's German patent filing they note that an electronic device, such as a mixed reality headset (HMD) may include one or more cameras or other sensors, such as a depth sensor, of which the depth of a scene can be determined. In one or more embodiments, each of the one or more cameras  may be a conventional RGB camera or a depth camera. ​Further, the cameras may include a stereo camera or other multi-camera system, a time-of-flight camera system, or the like that capture images from which depth information of a scene can be determined.

In some embodiments, the one or more cameras may be cameras used for eye tracking. The HMD device may allow a user to interact with XR environments. ​There are many different types of electronic systems that allow a person to perceive and/or interact with various XR environments.

Examples are head-mountable systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capability, windows with integrated display capability, displays formed as lenses intended to be placed on the eyes of a person (e.g., ​similar contact lenses). 

For more details, review Apple's German patent application number DE102022116074 titled "AUGENVERFOLGUNGSDATENFILTERUNG" (Translation: Eye Tracking Data Filtration).

Apple Inventors

• Sabine Webel: Software Engineering Manager. Came to Apple via acquisition of Metaio in 2015.

• Tobias Eble: Software Development Manager (former employer: Metaio)
• Hariprasad Puthukkootil Rajagopal: Sr. Software Developer (former employer: Metaio)

• Andreas Gapel: Software Development Manager (Former employer: Metaio)
• Ritesh Gangadhar Sholapur: Software Engineer. (Former employer Wevr)
• Jacob Wilson: Software Engineer