Today the U.S. Patent and Trademark Office officially granted Apple a patent that relates to future Vision related HMDs that include systems and methods that perform gaze tracking using combinations of ultrasound and imaging data. A second attribute of the eye is detected based on depth information from ultrasound based measurements of the eye, and a second location associated with the second attribute is determined in the 3D coordinate system based on the depth information and a 3D model of a portion of the eye. A gaze direction in the 3D coordinate system is determined based on the first location and the second location. In some implementations, user characteristics are identified (e.g., authentication) using ultrasonic biometric characteristics with imaging biometrics.

Hybrid Gaze Tracking System

Various implementations disclosed herein include hybrid devices, systems, and methods that perform gaze tracking for HMDs using combinations of ultrasound and imaging data.

Various implementations disclosed herein include devices, systems, and methods that perform gaze tracking using both imaging data and ultrasound data. In some implementations, such gaze tracking is more robust than either approach alone because imaging data is less affected by temperature, pressure, and air flow, and ultrasound data is less affected by stray light. In some implementations, different combinations of ultrasound data and light-based measurement data are used to effectively counter environmental noise, obtain desired accuracy, obtain desired power consumption rates, or combinations thereof. In some implementations, user authentication is improved by including additional ultrasonic biometric characteristics to imaging biometric characteristics.

In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of detecting a first attribute of an eye using light-based measurements in a first image of the eye, and determining a first location associated with the first attribute in a three dimensional (3D) coordinate system based on first depth information. In some implementations, a second attribute of the eye is detected based on second depth information from ultrasonic based measurements of the eye, and a second location associated with the second attribute is determined in the 3D coordinate system based on the second depth information from the ultrasound sensor and a 3D model of a portion of the eye. Then, a gaze direction in the 3D coordinate system is determined based on the first location and the second location.

In general, one innovative aspect of the subject matter described in this specification can be embodied in systems that include an image sensor configured to obtain images of the eye based on light reflecting off the eye, and an ultrasound sensor that includes at least one ultrasonic transmitter configured to transmit ultrasonic sound towards an eye of a user of an electronic device, and at least one ultrasonic receiver configured to receive an echo generated by interaction of the ultrasonic sound with the eye, wherein the ultrasound sensor is configured to obtain depth information of the eye in a 3D coordinate space. Then, a processor is configured to perform gaze tracking using the image sensor and the ultrasound sensor.

To review the full details of this invention, check out granted patent 12248626. The lead inventor is noted as Patrick Gill, Optical Research Scientist.