The U.S. Patent and Trademark Office officially published a series of 59 newly granted patents for Apple Inc. today. In this particular report we cover Apple's invention relating to eye-tracking in a future high-end Head Mounted Display Device. More specifically, the invention covers glint-assisted gaze tracking to provide high visual accuracy and clarity.
Virtual reality (VR) allows users to experience and/or interact with an immersive artificial environment, such that the user feels as if they were physically in that environment. For example, virtual reality systems may display stereoscopic scenes to users in order to create an illusion of depth, and a computer may adjust the scene content in real-time to provide the illusion of the user moving within the scene. When the user views images through a virtual reality system, the user may thus feel as if they are moving within the scenes from a first-person point of view.
Similarly, mixed reality (MR) or augmented reality (AR) systems combine computer generated information (referred to as virtual content) with real world images or a real-world view to augment, or add content to, a user's view of the world.
The simulated environments of VR and/or the mixed environments of MR may thus be utilized to provide an interactive user experience for multiple applications, such as applications that add virtual content to a real-time view of the viewer's environment, interacting with virtual training environments, gaming, remotely controlling drones or other mechanical systems, viewing digital media content, interacting with the Internet, or the like.
A Head Mounted Display device must therefore deliver deadly accurate data to the users view so that the user is one with the VR or AR environment. Having incorrect augmented reality imagery placed in a scene would be undesirable and inaccuracy viewed in a first-person shooter would be distracting to the user being unable to aim and shoot with pinpoint precision.
Apple's granted patent covers methods and apparatus for gaze tracking in virtual reality (VR) or augmented reality (AR) devices. Embodiments of a VR/AR device such as a headset, helmet, goggles, or glasses (referred to as a head-mounted display (HMD)) are described that include a display mechanism (e.g., left and right near-eye display panels) for displaying frames including left and right images in front of a user's eyes to thus provide 3D virtual views to the user.
The HMD may include left and right optical lenses (eye lenses) located between the display and the user's eyes. For AR applications, the HMD may include or be coupled to one or more external video cameras that capture video of the user's environment for display. The HMD may include a controller component that may, for example, render frames for display to the left and right displays. Alternatively, the controller component may be implemented by an external device that is coupled to the HMD via a wired or wireless connection.
A gaze tracking system is included in the HMD for detecting position and movement of the user's eyes. The gaze tracking system may include at least one eye tracking camera (e.g., infrared (IR) or near-IR (NIR) cameras) positioned at each side of the user's face, and illumination sources (e.g., IR or NIR light sources such as an array or ring of LEDs) that emit light (e.g., IR or NIR light) towards the user's eyes.
The eye tracking cameras may be pointed towards the user's eyes to receive reflected IR or NIR light from the light sources directly from the eyes, or alternatively may be pointed towards "hot" mirrors located between the user's eyes and the display panels that reflect IR or NIR light from the eyes to the eye tracking cameras while allowing visible light to pass. The gaze tracking system may capture images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps)), analyze the images to generate gaze tracking information, and communicate the gaze tracking information to the controller component.
In the gaze estimation process, a 3D cornea center estimation process estimates the center of the user's cornea in 3D space based on the detected glints and LED correspondences and user calibration data representing the specific user's eye parameters.
A 3D pupil center estimation process estimates the center of the user's pupil in 3D space based on the detected pupil ellipse, the user calibration data, and output of the cornea center estimation process.
An optical axis reconstruction process reconstructs the optical axis of the user's eye (the axis connecting the cornea center and the pupil center) in 3D space based on output of the cornea center estimation process and the pupil center estimation process.
A visual axis reconstruction process reconstructs the visual axis of the user's eye (the axis connecting the fovea and the cornea center) in 3D space based on output of the optical axis reconstruction process and the user calibration data. A distorted display point estimation process estimates a point on the HMD display (the point of gaze) based on the output of the visual axis reconstruction process and the device-specific HMD calibration data.
Apple's patent FIGS. 3A, 3B, 3C and 3D below show side and front views of example HMDs that implement a gaze tracking system.
Apple's patent FIGS. 9A through 9C graphically illustrate glint-LED matching in image space; FIGS. 11A and 11B show example results of a glint matching in image space method compared to results when a glint geometric matching method is applied in 3D space to detect and correct potential mismatches using the glint matching in image space method.
Apple's patent FIG. 12A below illustrates a model of a human eye in relation to a display of an HMD; FIG. 12B illustrates a mathematical model for pupil center estimation; and FIG. 12C illustrates a mathematical model for cornea center estimation
Apple's granted patent was originally filed in Q3 2018 and published today by the US Patent and Trademark Office.
Han Lin Kuen: Computer Vision Algorithm Engineer
Tsin Yanghai: Senior Engineer leader in computer vision and machine learning.
Granted Patent #2: Determining a Gaze Point on a Three-Dimensional Object
Apple was also granted another major "gaze tracking system" patent today under number 10,699,439. This is yet another in-dept patent that Apple inherited when they acquired the German company SMI SensoMotoric Instruments in June 2017.