Apple Patent Reveals Eye-Tracking & Near Infrared Camera Systems behind their Future Mixed Reality Headset
Last month the U.S. Patent Office published four major Head Mounted Display patent applications from Apple. These four patent pending inventions (one, two, three and four) in my view represented the patents of the decade for Apple. Collectively the patents covered a mixed reality headset that could be used in autonomous vehicles, as a desktop replacement or a next-gen mobile computer, capable of working with 3D documents, deliver exceptional first-person gaming and virtual reality experiences using a special predictive and foveated display.
Today the U.S. Patent Office published Apple's fifth invention supporting Apple's next-gen mixed reality headset that specifically dives into the advanced eye-tracking system that will deliver these next generation experiences using hot mirrors and a Near Infrared (NIR) camera system.
Apple notes in their patent filing that virtual reality (VR) allows users to experience and/or interact with an immersive artificial environment, such that the user feels as if they were physically in that environment. For example, virtual reality systems may display stereoscopic scenes (3D scenes) to users in order to create an illusion of depth, and a computer may adjust the scene content in real-time to provide the illusion of the user moving within the scene.
When the user views images through a virtual reality system, the user may feel as if they are moving within the scenes from a first-person point of view. Similarly, mixed reality (MR) combines computer generated information (referred to as virtual content) with real world images or a real world view to augment, or add content to, a user's view of the world.
The simulated environments of VR and/or the mixed environments of MR may be utilized to provide an interactive user experience for multiple applications, such as applications that add virtual content to a real-time view of the viewer's environment, interacting with virtual training environments, gaming, remotely controlling drones or other mechanical systems, viewing digital media content, interacting with the Internet, or the like.
An eye tracker is a device for estimating eye positions and eye movement. Eye tracking systems have been used in research on the visual system, in psychology, psycholinguistics, marketing, and as input devices for human-computer interaction. In the latter application, typically the intersection of a person's point of gaze with a desktop monitor is considered.
Apple's latest HMD invention covers various embodiments of methods and apparatus for eye tracking in virtual and mixed or augmented reality (VR/AR) applications are described.
A VR/AR device such as a headset, helmet, goggles, or glasses (referred to in their patent filing as a head-mounted display (HMD)) is described that includes a display (e.g., left and right displays) for displaying frames including left and right images in front of a user's eyes to thus provide 3D virtual views to the user. The HMD may include left and right optical lenses (referred to herein as eyepieces) located between the display and the user's eyes. The eyepieces form a virtual image of the displayed content at a design distance which is typically close to optical infinity of the eyepieces.
The HMD may include an eye tracking system for detecting position and movements of the user's eyes. The eye tracking system may include at least one eye tracking camera (e.g., near-IR (NIR) cameras) positioned at each side of the user's face and pointed towards the eye-facing surfaces of the respective eyepieces, an illumination source (e.g., an NIR light source) that emits light (e.g., NIR light) towards the user's eyes, and hot mirrors located between the eye-facing surfaces of the eyepieces and the user's eyes.
Positioning the hot mirror surfaces at or near the eye-facing surfaces of the eyepieces allows the eye tracking cameras to be placed at the sides of the user's face (e.g., at or near the user's cheek bones) without having to image through the eyepieces as presented in Apple's patent FIG. 3 below.
In some embodiments, the light sources of the HMD emit NIR light to illuminate the user's eyes. A portion of the NIR light is reflected off the user's eyes to the hot mirrors located at or near the eye-facing surfaces of the eyepieces of the HMD. The hot mirrors reflect at least a portion of the NIR light, while allowing visible light to pass. The NIR cameras, for example located at or near the user's cheek bones capture images of the user's eyes reflected by the hot mirrors.
Images captured by the eye tracking system may be analyzed to detect position and movements of the user's eyes, or to detect other information about the eyes such as pupil dilation. For example, the point of gaze on the display estimated from the eye tracking images may enable gaze-based interaction with content shown on the near-eye display of the HMD. Other applications may include, but are not limited to, creation of eye image animations used for avatars in a VR/AR environment.
Apple's patent FIG. 1A below illustrates an eye tracking systems for VR/AR head-mounted displays (HMDs); FIG. 4 is a block diagram illustrating components of an example VR/AR system that includes an eye tracking system.
In some embodiments, Apple's HMD may also include one or more interfaces (e.g., a Bluetooth technology interface, USB interface, etc.) configured to communicate with an external device via a wired or wireless connection. In some embodiments, at least a part of the functionality described for the controller may be implemented by the external device which could be any type of computing system or computing device, such as a desktop computer, MacBook, iPad, iPhone, game controller and more.
Apple's patent application was filed back in Q4 2017. Considering that this is a patent application, the timing of such a product to market is unknown at this time.
A Few of the Engineers behind this Invention
A few of the inventors behind this invention include: Ms. Berkner is a Senior Engineer Camera Incubation who came to Apple via Ricoh where she was the director of California R&D; Mr. Kenichi is an Optical Engineer who was an Optical Engineer and Senior Scientist at Cannon; Mr. Kim is an Apple Camera Prototyping Engineer who was a PhD intern at Samsung; and Mr. Sauers, Product Designer.
Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of patent application should be read in its entirety for full and accurate details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.