Apple Invents an Eye Tracking System for the Creation of Eye Image Animations associated with next-gen Avatars in a VR environment
On Thursday the US Patent & Trademark Office published a patent application from Apple that relates to an eye tracking system which is going to play a key role in any future mixed reality headset. We've posted at least a dozen patent related reports covering different aspects of a future eye tracking / eye gaze system over the last 15. Below are just a few of the reports that we've posted.
01: Apple Introduces the use of Neural Networks to advance Eye and Gaze Tracking in a Future HMD
02: Another Dimension of Apple's Eye Tracking Technology reveals the use of Biometrics and Machine Learning
03: Apple Reveals Advanced Controls for their Future Head Mounted Device that will include Eye Gaze, Siri, Touch & more
04: Apple is Considering the use of Eye-Tracking & Gaze Controls with iDevices & Head Mounted Display Devices
05: An AR Smartglasses patent from Apple Surfaced today covering an advanced Eye Tracking System
Apple's latest eye tracking invention covers dual eye tracking cameras and an illumination source that emits infrared light towards the user's eyes, and diffraction gratings located at the eyepieces. Our report covers only the basics of the system. For greater details you'll have to review the patent filing that we link to at that bottom of the report.
So, what are Apple engineers focusing on in this invention that could be coming to Apple's future HMD? It starts with patent FIG. 9 below which illustrates a side view of an example Mixed Reality headset (HMD #200) that implements an eye tracking system. The eye tracking system may, for example, be used to track position and movement of the user's (#290) eyes (#292).
In some embodiments, the eye tracking system may instead or also be used to track dilation of the user's pupils, or other characteristics of the user's eyes.
Information collected by the eye tracking system may be used in various MR (mixed reality) system functions. MR = AR + VR content. Apple notes that "Other applications of the eye tracking information may include, but are not limited to, creation of eye image animations used for avatars in an MR environment."
Technically, Apple wants to bring eye tracking to be used in context of a Memoji or other type of avatar-like character you're playing in a game or other VR environment. Today Apple's iPhone does that well, and in the future, the next-gen eye tracking system will allow for this feature to be used in a VR headset.
As another example, in some embodiments, the information collected by the eye tracking system may be used to adjust the rendering of images to be projected, and/or to adjust the projection of the images by the projection system of the HMD #200, based on the direction and angle at which the user's eyes are looking.
As another example, in some embodiments, brightness of the projected images may be modulated based on the user's pupil dilation as determined by the eye tracking system.
Apple's patent FIG. 11 below is a block diagram illustrating components of an example MR system that includes an eye tracking system.
What Apple has planned for IKinema technology is unknown at this time but it could take the Memoji/Avatar feature to the next level for VR environments and somehow eye tracking set into a headset could play a role.
For finer details of Apple's patent, check out application number 20210041948. Apple's 23 patent claims that are located towards the top half of the patent filing will provide you with an overview of the technical aspects of the invention that Apple thinks are most important and wish to protect. Considering that this is a patent application, the timing of such a product to market is unknown at this time.
Side Note: Many times, I've been asked "Where's the iPhone patent?" My response is always the same, there's no such thing. Any major device that Apple has ever brought to market (iPhone, iPad, Apple Watch, Macs, AirPods Pro etc.) has derived from a suit or family of patents over a period of time. The many patents in the suit cover contributions from different Apple engineering teams that are each focused on very specific parts of an invention like an iPhone, from audio to cameras, to sensors, to materials and so forth.
Likewise, there is no one patent for an eye tracking system and this is why we'll continue to find new patents on this subject matter. Various Apple teams present different ways to implement an eye tracking system that could compliment or contradict past patents.
At some point in time, Apple's Senior VP's collaboratively decide which patented inventions provide the best solution for a particular end device and viola, version one comes to market while their engineering teams continue to push new ideas for yet future versions of a device. It's an ongoing process.