On September 28, 2023, a patent application from Apple was published in Europe relating to a vision system or new Face ID system for smartglasses and Vision Pro that will allow Face ID to work with the cameras on and in the HMD to unlock companion devices such as Apple Watch, iPhone, iPad and Macs.   

Systems and Methods for User Authenticated Devices

Apple's granted patent addresses the challenge resulting from a head-mounted device, such as future smart glasses, partially or completely occluding a user's eyes with IR filters, and thereby preventing the user from unlocking a companion device, such as an iPhone using Face technology.  

According to one example, the present invention addresses and overcomes these challenges by using an on-board vision system of the head-mounted device to determine, based on a location of the companion device and a gaze of the user, whether the user is looking at the companion device, and thus intending to unlock the device. 

In an example operational sequence, a user wearing a head-mounted device/smart glasses with IR filters desires to unlock a companion device, such as a smartphone or tablet, using facial recognition. The user can orient the companion device such that a vision system of the companion device locates the user's face and begins a facial recognition sequence. Because the user's eyes are occluded by the IR filters, the companion device may be able to only complete a partial match of the user's face, insufficient to unlock the companion device.

The head-mounted device and the companion device can exchange electronic communications or transmissions to prompt a vision system of the head-mounted device to determine or detect an orientation of a gaze of the user, for example, via a user-facing camera. The orientation of a gaze or gaze direction of the user can correspond to what the user is looking at or the line of sight of the user.

In some examples, this can be referred to as the eye pose (e.g., a determination of the direction in which the eye is looking). The pose or orientation of the eye can be determined based on the positions of ocular characteristics, such as the pupil, iris, cornea, glints, and other ocular characteristics. The position of the ocular characteristics can be determined relative to the head-mounted device, the eyelids or face of the user, or relative to the natural static direction of the eye.

Further, the vision system of the head-mounted device and/or of the companion device can determine or detect a location of the companion device relative to the head-mounted device, for example, with a second, outward-facing camera.

In some examples, a single camera can both detect the orientation of the user's gaze and the location of the companion device. Once a gaze direction or orientation of the user and a relative location of the companion device are determined, the system can determine whether the user is looking at the companion device. That is, the system can determine whether the orientation of the user's gaze satisfies a condition based in part on the location of the companion device. The condition satisfied by the orientation of the user's gaze can be that the user's gaze is directed at the companion device.

In some examples, the condition can thus be based at least in part on a location of the companion or proximate device. In some examples, the condition is satisfied if the gaze direction is oriented toward the position of the companion device. In some examples, the orientation of the gaze satisfies the condition when the orientation of the user's gaze is within a predetermined degree threshold of the location of a display of the proximate or companion device.

For example, if the orientation of the gaze is within 5 degrees of any portion of the companion device. In some examples, it is determined that the user is looking at the companion device if the orientation of the gaze is within 1 degree of any portion of the companion device.

If it is determined that the user is looking at the companion device, the head-mounted device can transmit a signal to the companion device. The signal can contain informational data. For example, the head-mounted device can provide authentication credentials to unlock the companion device.

Apple's patent FIG. 2C illustrates smartglasses with an on-board vision system. While the patent emphasizes smartglasses, the HMD could be a future version (or the current version)I of Vision Pro; FIG. 2B shows a side view of smartglasses with a frame; FIG. 2A Shows a system for user authentication for companion devices (Apple Watch, iPhone, Mac).

Apple's patent FIG. 2D above shows example gaze and image tracking capabilities of Smart Glasses 100c. Specifically, FIG. 2D illustrates an example in which the user is not looking at the companion device #200.

Apple's patent FIG. 2E shows an example in which the user is looking at the companion device #200. In response to determining or detecting that the orientation of the user's gaze #241 is within a predetermined degree threshold of the location of the companion device, it can be determined that the user is looking at the companion device indicating that the user desires to unlock the companion device.

Apple's patent FIGS 3 & 4 above illustrate flow diagrams of example processes for user authentication.

For more details, review Apple's patent application 20230308873 published on September 28, 2023. The European filing ID (presented below) shows that the owner of the patent is in fact Apple in contrast to the U.S. filing that was filed under the names of Apple's inventors/engineers. This is done to keep the invention under cover as long as they can. When the patent is granted, Apple is then forced to identify themselves as the invention's Intellectual Property owner.

Whether this will be a feature for Vision Pro in 2024 or further down the line is not known at this time.