Today the U.S. Patent and Trademark Office officially published a series of 58 newly granted patents for Apple Inc. In this particular report we briefly cover four Vision Pro related inventions that relate to realistic avatar creation for FaceTime, Retinal Eye Tracking and more. And as always, we wrap up this week's granted patent report with our traditional listing of the remaining granted patents that were issued to Apple this week.

Generating Facial Expressions for Vision Pro

One of Apple's granted patents today relates to Vision Pro's realistic Avatar generation as noted in our cover graphic. The patent covers generating an expressive avatar includes obtaining an expressive image of a face, and determining at least one target levels of detail for a representation of the face. A trained convolutional neural network model is applied to the expressive image of the face to obtain a mesh representation of the face at each of the at least one target levels of detail, wherein each convolution in the trained convolutional neural network model provides a unique level of detail, and an avatar is generated based on the mesh representation of the face at least of the at least one target levels of detail.

Apple further notes that an XR module may be configured to support extended reality services, such as rendering an avatar, for example, depicting a face. The avatar may be rendered according to an output mesh and additional data, such as head pose, lighting condition, and a view vector.

Apple's patent FIG. 1 above illustrates an example diagram of a technique for generating facial expression meshes using a constrained neural network. Review granted patent 12073501 for details.

3D Representation Adjustments for Communication Sessions

This second granted patent supports the first granted patent presented above in nature. The particular patent covers various implementations disclosed herein include devices, systems, and methods that provide a view of an adjusted position of a three-dimensional (3D) representation of another user (e.g., an avatar) during a communication session. At a first device, 3D representation data corresponding to at least a portion of a second user of a second device (e.g., a sender) is obtained during a communication session. Offset information (e.g., a vertical offset value) corresponding to an offset between a camera viewpoint position and an eye viewpoint position associated with the second device is obtained. A position of a first portion of a 3D representation of the second user (e.g., head and torso) is adjusted relative to a 3D environment associated with the first device. A view that includes the first portion of the 3D representation of the second user in the 3D environment is provided during the communication session.

Apple's patent FIG. 4 below illustrates an offset between a camera viewpoint position and an eye viewpoint position associated with a head mounted device (HMD) during a communication session.

As illustrated in FIG. 4, the HMD includes an inward facing camera (sensor #425) that captures images of the face, and in particular the eyes and eye gaze (e.g., eye viewpoint #412), of the user #430 as the user is looking towards the eyes of the representation #440 of the other user (e.g., represented by eye contact line #414). The HMD further includes an outward facing camera (sensor #420) that captures a camera point of view (e.g., camera viewpoint #422) that is used by the HMD to display to the user what he or she thinks is the actual point of view.

For more details, review granted patent 12073530.

Vision Pro Hardware Overview

Two weeks ago Patently Apple posted an IP report titled "Today Apple won 43 patents covering one that presents everything you ever wanted to know about the construction of Vision Pro.' If that wasn't enough, Apple was granted a second massive Vision Pro hardware overview with dozens of detailed patent figures. For full details, review granted patent 12075597.

Multiple Gaze Dependent Illumination Sources For Retinal Eye Tracking

In Apple's patent background they note that retinal imaging-based tracking systems generally have both an illumination source to direct light towards the retina and an image sensor to generate images including light reflected from the retina. Drawbacks of such tracking systems include the relatively significant amount of power used by the illumination sources and unwanted reflections (glare) from eye surfaces other than the retina, e.g., cornea, crystalline lens, etc.

Apple's granted patent 12073018 covers various implementations that include devices, systems, and methods that provide retinal imaging-based gaze tracking. In some implementations, a user's gaze is tracked based on a retinal imaging technique that selectively illuminates a subset of multiple light sources to direct light towards one or more portions of the user's retina. In some implementations, the subset of the light sources is used in retinal imaging to provide better gaze tracking by reducing glare, reducing image saturation, reducing imaging computation, or reducing energy consumption.

Apple's patent FIG. 3 below is a block diagram that illustrates an exemplary tracking system that selectively uses a subset of multiple light sources to illuminate one or more portions of a retina for gaze tracking; (5) FIG. 4 is a diagram that illustrates glare from exemplary light sources used to illuminate a retina; and   FIG. 6 is a diagram that illustrates exemplary light sources near an image sensor used to illuminate a retina.

This Week's Remaining Granted Patents