The U.S. Patent and Trademark Office officially published a series of 67 newly granted patents for Apple Inc. today. In this particular report we cover two eye tracking HMD systems and point out two additional patents covering HomePod mini and Apple TV being able to detect the presence of a user to turn on the TV or even transfer a FaceTime call from an iPhone/iPad to the TV automatically.  And as always, we wrap up this week's granted patent report with our traditional listing of the remaining granted patents that were issued to Apple today.

HMD: Low-Power Eye Tracking System

Apple's first HMD granted patent covers various embodiments of methods and apparatus for eye tracking in virtual and mixed or augmented reality (VR/AR) applications. A VR/AR system may include a device such as a headset, helmet, goggles, or glasses (referred to herein as a head-mounted device (HMD)) that includes a display (e.g., left and right displays) for displaying frames including left and right images in front of a user's eyes to thus provide three-dimensional (3D) virtual views to the user.

A VR/AR system may also include a controller. The controller may be implemented in the HMD, or alternatively may be implemented at least in part by an external device (e.g., a computing system) that is communicatively coupled to the HMD via a wired or wireless interface. The controller may include one or more of various types of processors, image signal processors (ISPs), graphics processing units (GPUs), coder/decoders (codecs), and/or other components for processing and rendering video and/or images. The controller may render frames (each frame including a left and right image) that include virtual content based at least in part on the inputs obtained from cameras and other sensors on the HMD, and may provide the frames to a projection system of the HMD for display.

The VR/AR system may include an eye tracking system (which may also be referred to as a gaze tracking system). Embodiments of an eye tracking system for VR/AR systems are described that include at least one eye tracking camera (e.g., infrared (IR) cameras) positioned at each side of the user's face and configured to capture images of the user's eyes.

The eye tracking cameras, for example located at or near edges of the HMD display panel(s), capture images of the user's eyes from the IR light reflected off the eyes. Images captured by the eye tracking system may be analyzed by the controller to detect features (e.g., pupil), position, and movement of the user's eyes, and/or to detect other information about the eyes such as pupil dilation.

A key to providing accurate eye tracking is knowing the location of the user's eyes with respect to the eye tracking cameras. In some embodiments of an eye tracking system, to accurately determine the location of the user's eyes with respect to the eye tracking cameras, the controller may execute an algorithm that performs a three-dimensional (3D) reconstruction using images captured by the eye tracking cameras to generate 3D models of the user's eyes. The 3D models of the eyes indicate the 3D position of the eyes with respect to the eye tracking cameras, which allows the eye tracking algorithms executed by the controller to accurately track eye movement.

Apple's patent FIG. 13 below illustrates an example alternative VR/AR device (#3000) that includes an eye tracking system with head odometers (#3042) that detect movement of the device with respect to the user's eyes (#3092) and eye odometers (#3044) that track movement of the eyes in intervals between the processing of frames captured by the eye tracking cameras (#3040).

Apple's patent FIG. 11 above is a flowchart of an eye tracking method in which head odometers are used detect movement of the HMD with respect to the user's eyes and eye odometers are used to track movement of the eyes in intervals between the processing of frames captured by the eye tracking cameras.

For more details, review Apple's granted patent 11,307,659. Eye or Gaze Tracking technology is going to play a major role in Apple's future mixed reality headset and there's been many, many patents on this like these: 01, 02, 03, 04, 05 and 06; with more in our archives.

HMD: Modifying Mixed Reality Content to invoke a Target User State

In September 2020, Patently Apple posted an IP report titled "Apple Envisions the Workplace & Auditoriums adopting a next-gen Communications System delivering AR/VR Content to HMDs through Walls." One of the patent figures from that patent is presented below.

Flash-forward to today and we're now covering Apple's second HMD granted patent of the day wherein we see in patent FIG. 6 below, Apple is once again working on the same concept that we covered in 2020. In 2020, Apple's engineers referred to a "Relay Dock" whereas below, another Apple team refers to the Relay Dock as "remote input devices" or at one point "a controller" that could be a company server or cloud server. Apple's patent FIG. 6 below is a block diagram of an example operating architecture #600

Although there are similarities to the 2020 patent, today's granted patent adds several layers of features. For instance, in some implementations, Apple notes that the method includes obtaining sensor information associated with the user, wherein the sensor information corresponds to one or more physiological measurements of the user; determining, using a qualitative mood classifier, a resultant state of the user based on the sensor information associated with the user; and updating the user model based at least in part on the resultant state of the user and the modified CGR content.

In some implementations, the one or more physiological measurements of the user include at least one of eye tracking information, pupil dilation diameter information, body pose characteristics, speech characteristics, heart rate, glucose level, and blood oximetry. In some implementations, the method further includes determining the resultant state of the user by determining whether a user provided informed consent to monitor one or more physiological modalities associated with the user information.

If this subject matter interests you, Apple's granted patent 11,307,650 is deeply detailed and you could find it here.

Other Granted Patents of Interest Today

Other granted patents of interest published today include one for a "Compact Speaker" which translates to HomePod mini (see patent 11, 310, 585); another is titled "User detection by a computing device." For a quick review of that patent, you could check out our original 2017 report for details here or Apple's full granted patent 11,307,657 here. One of the patent figures from that patent is presented below.

Today's Remaining Granted Patents