Apple wins a couple of new Eye Tracking System Patents covering Future Glasses and AV/AR Headset
Today the U.S. Patent and Trademark Office officially granted Apple two eye tracking patents. The first relates to a method for performing a calibration procedure for calibrating an eye tracking device. One of Apple's inventors is listed as Walter Nistico, Engineer Manager, Deep Learning and Computer Vision. The second patent relates to a VR/AR head-mounted display (HMD) that includes a gaze tracking system for detecting position and movement of the user's eyes.
In the first granted patent, Apple notes that eye tracking devices are configured to track persons, especially their eyes, their point of regard or gaze direction. As certain properties of human eyes are different from person to person, usually a calibration procedure is performed to calibrate the eye tracking device, so that the accuracy of the determined gaze or gaze point can be improved.
The main advantage of this invention is, that the execution of the calibration procedure does not follow a fixed procedure but instead can be adapted in dependency of at least one determined control parameter. Especially, this is based on the finding that the captured gaze data provide information, which advantageously can be used to control the execution of the calibration procedure to provide better adaption, especially adaption to the properties and capabilities of the user performing the calibration.
Generally, the system comprises an eye tracking device and a processing unit, which may or may not be included within the eye tracking device, and which controls the stimulus and performs all the necessary computations for the calibration. Further, the system comprises means to show the calibration stimulus to a user in form of the display device, which can be implemented for example through a monitor screen or a head mounted display or a virtual reality headset, etc.
In particular, the system can be implemented as a head mounted system comprising the eye tracking device, which comprises one or more cameras, which are integrated in the head mounted system. This can be for example in the form of glasses, or a helmet, or any other fixture on a user's head.
Apple's patent FIG. 1 below shows a schematic illustration of an eye tracking system (#10), which is configured as a remote system according to a first embodiment of the invention.
The eye tracking system comprises an eye tracking device (#12), which comprises an imaging device which is configured as at least one camera (#14) and capable of capturing images of at least one eye (#16) of a user (#18).
Moreover, the eye tracking system 10 comprises a display device (#20), which is configured as a monitor, and which provides in this example a 2D display area 22. Furthermore, the eye tracking system comprises a processing unit #24, which can be integrated into the display device or integrated into the eye tracker, or also be configured as a separate entity as a standalone processing unit and be communicatively coupled to the display device to control the display device and to the eye tracker to analyze the captured images.
Apple's patent FIG. 2 above shows a schematic illustration of another example of an eye tracking system, which is configured as a head mounted system according to an embodiment of the invention. Especially the eye tracking system is implemented in form of augmented reality or virtual reality glasses.
This is definitely a patent that engineers and/or developers will love to explore. It's not a consumer-friendly read. For more details, review Apple's granted patent 10,976,813.
Focusing for Virtual and Augmented Reality Systems
A second eye tracking system patent was granted to Apple today. Patent 10,979,685 titled "Focusing for virtual and augmented reality systems" The sole inventor listed on the patent is Ammon Silverstein, Senior Director at Apple who is a leader of architecture in camera image processing, image analysis and neural compute.
Apple's granted patent covers methods and apparatus for focusing in virtual reality (VR) or augmented reality (AR) devices based on gaze tracking information are described. Embodiments of a VR/AR head-mounted display (HMD) may include a gaze tracking system for detecting position and movement of the user's eyes.
For AR applications, gaze tracking information may be used to direct external cameras to focus in the direction of the user's gaze so that the cameras focus on objects at which the user is looking.
For AR or VR applications, the gaze tracking information may be used to adjust the focus of the eye lenses so that the virtual content that the user is currently looking at on the display has the proper vergence to match the convergence of the user's eyes.
Apple's patent FIG. 2 below shows a side view of an example HMD that implements a gaze tracking system; FIG. 1E illustrates focusing eye lenses in VR HMDS based at least in part on gaze tracking information.
Moments after posting today's report I discovered a third eye tracking related patent titled "Method and device for eye tracking with content-embedded glints."
In one implementation, Apple's invention cover a method includes: synthesizing an AR/VR content stream by embedding a plurality of glints provided for eye tracking into one or more content frames of the AR/VR content stream; displaying, via the one or more AR/VR displays, the AR/VR content stream to a user of the HMD; obtaining, via the image sensor, light intensity data corresponding to the one or more content frames of the AR/VR content stream that include the plurality of glints, wherein the light intensity data includes a projection of an eye of the user of the HMD having projected thereon the plurality of glints; and determining an orientation of the eye of the user of the HMD based on the light intensity data.
For more on this, review granted patent 10,976,817.