In June 2017 Patently Apple posted a report titled "Apple may have acquired a Company Specializing in Eye Tracking Glasses covering Augmented Reality & More." Then in January 2018 Patently Apple posted a report titled "An Apple Patent Inherited from SensoMotoric on Advanced Eye Tracking for Smartglasses was Published Today."

Today the US Patent & Trademark Office published another patent application from Apple that was inherited from SensoMotoric that relates to a Head-Mounted Display and/or smartglasses and method for gaze endpoint determination, in particular for determining a gaze endpoint of a subject on a three-dimensional object in space. One of the inventors listed on this patent is Tom Sengelaub, Engineering Manager at Apple.  Sengelaub was the Project Lead Virtual Reality Engineering Manager at SensoMotoric.

The invention covers a system for determining the gaze endpoint of a subject, the system comprising: an eye tracking unit adapted to determine the gaze direction of one or more eyes of the subject; a head tracking unit adapted to determine the position comprising location and orientation of the head and/or the eye tracking unit with respect to a reference coordinate system; a 3D scene structure representation unit, that represents a real-world scene and objects contained in the scene by representing the objects of the real-world scene through their 3D position and/or their 3D-structure through coordinates in the reference coordinate system to thereby provide a 3D structure representation of the scene; a calculating unit for calculating the gaze endpoint based on the gaze direction, the eye tracker position and the 3D scene structure representation, and/or for determining the object in the 3D scene the subject is gazing at based on the gaze direction, the eye tracker position and the 3D scene structure representation

By using a 3D representation, an eye tracker and a head tracker there can be determined not only a gaze point on a 2D plane but also an object the subject is gazing at and/or the gaze endpoint in 3D.

According to one embodiment the system comprises a module for calculating the gaze endpoint on an object of the 3D structure representation of the scene, wherein said gaze endpoint is calculated based on the intersection of the gaze direction with an object in the 3D structure scene representation.

The intersection of gaze direction with the 3D representation gives a geometrical approach for calculating the location where the gaze "hits" or intersects the 3D structure and therefore delivers the real gaze endpoint. Thereby a real gaze endpoint on a 3D object in the scene can be determined.

According to one embodiment the system comprises a module for calculating the gaze endpoint based on the intersection of the gaze directions of the two eyes of the subject, and/or a module for determining the object the subject is gazing at based on the calculated gaze endpoint and the 3D position and/or 3D structure of the objects of the real world scene.

By using the vergence to calculate the intersection of the gaze direction of the eyes of the subject there can be determined the gaze endpoint. This gaze endpoint can then be used to determine the object the user is gazing at.

The invention covers a head-mounted display device or smartglasses that includes an eye tracker and a scene camera that is a stereo camera system.

Apple's patent FIG. 2 below schematically illustrates an offline gaze endpoint determination system according to an embodiment of the invention.

Apple's patent FIG. 3 below schematically illustrates an online gaze endpoint determination system according to an embodiment of the invention.

Apple's patent application that was published today by the U.S. Patent Office was re-filed by Apple with tweaks to its patent claims back in Q1 2019. The original patent was published by SensoMotoric in Europe both in 2013 and 2015. Considering that this is a patent application and one that Apple inherited, the timing of such a product to market is unknown at this time.

Apple's first granted patent from SensoMotoric was issued in July 2019 and you can review our report on it here.

About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Apple IP 20190272650