Last Friday Patently Apple posted a report titled "Samsung Invents Augmented Realty Smartglasses that will compete with Huawei's Upcoming Gentle Monster AR Glasses." Towards the end of that report we asked the question "Could Apple be in the AR Smartglasses Race?" The answer is certainly – even with a wishy-washy rumor set into motion by a Digitimes report hidden behind a paywall. Today a major smartglasses patent was granted to Apple as their work continues on this project.
The U.S. Patent and Trademark Office officially published a series of 60 newly granted patents for Apple Inc. today. In this particular report we briefly cover one of Apple's head mounted eye tracking device inventions for a future smartglasses device. The invention came by way of Germany's SMI SensoMotoric Instruments which Apple Acquired back in July 2017.
The CTO and head of engineering for that company, Walter Nistico, is now with Apple as Engineering Manager – Deep Learning and Computer Vision; Martin Haller was technical lead for automated semantic gaze mapping and now with Apple as Computer Vision Engineer; and Tom Sengelaub who was project lead for Virtual Reality is now with Apple as an Engineering Manager.
Two of Sengelaub's presentations that were given at Augmented Word Expo 2016 and 2017 are presented below.
Eye-Tracking Powered Hands-free Interaction & Personalization
Experiences in Bringing Eye Tracking to Augmented Reality
Head Mounted Eye Tracking Device
Apple's newly granted patent is titled "Head Mounted Eye Tracking Device and method for providing Drift Free Eye Tracking through a Lens System."
The invention relates to a head mounted eye tracking device for determining at least one first feature of at least one eye, wherein the head mounted eye tracking device comprises at least one capturing device for capturing light reflected by at least one eye of a user and at least one optical component capable of altering a propagation direction of light passing through the optical component. Furthermore, the invention relates to a method for determining at least one first feature of at least one eye of a user by means of a head mounted eye tracking device.
Apple's patent FIG. 1 below is a schematic illustration of a head mounted eye tracking device according to a first embodiment. Generally, the head mounted eye tracking device 10a comprises a capturing device C, which can comprise one or more cameras or sensors for taking pictures of the eye 12 of a user wearing the head mounted eye tracking device 10a. Furthermore, eye tracking device #10a comprises an optical component #14 which can comprise one or more lenses, prisms or other optical elements.
Apple's patent FIG. 2 below a schematic illustration of a head mounted eye tracking device according to a second embodiment. In this embodiment an optical component #14 comprises a free-form lens E and light captured by the capturing device C propagates from the eye #12 through the free-form lens E, is reflected by the hot mirror M and then captured by the capturing device C.
Also other methods like raytracing, reconstructing the eyes in virtual coordinate systems, undistorting the camera image, using a virtual camera and/or reconstructing the gaze on a virtual stimulus plane can be used and discussed in the granted patent.
In patent FIG. 3 below we're able to see a schematic illustration of the principle of raytracing for use in a head mounted eye tracking device #10a and #10b, especially for taking into account the optical properties of the optical component #14.
An idea of this method is to trace the rays back from the capturing unit C, e.g. a camera, or of a light source, which is denoted with a reference sign B in FIG. 3, into the direction Dir1 of the detected features, that are represented by the observed point P1 in FIG. 3, until they hit the optical component #14. Then, the ray at the optical component is refracted and one obtains a new outgoing ray, especially a refracted ray, which can be described by a point P2 on that ray and its new direction Dir2.
Patent FIG. 4 above shows a schematic illustration of reconstructing the eye in virtual coordinate system for use in a head mounted eye tracking devices 10a & 10b.
Apple's granted patent 10,354,136 was published today by the US Patent and Trademark Office. It was acquired by Apple in 2017.
About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.