In the last week Patently Apple has posted two patents relating to Apple Vision Pro's enrollment process in creating a realistic avatar of a user for FaceTime calls. Today the US Patent & Trademark Office published a patent application from Apple that continues the details of the process, and more importantly, reveals how a user will be able to capture and enroll images of their face and hands into an Apple Watch via the sensors/cameras on the backside of Apple Watch.

Once enrolled, the user will be able to join a FaceTime call and be seen by others in the form of a realistic avatar, just like with Vision Pro. In real-time the users mouth will move in sync with the user's voice. Is it me, or is this a really cool concept? If it works on Vision Pro today, then bringing this invention to Apple Watch would seem to be something we could expect in the next few years.   

Apple states in their patent that there is a need for electronic devices with improved methods and interfaces for generating and/or displaying representations of users. Such methods and interfaces may complement or replace conventional methods for generating and/or displaying representations of users.

During an enrollment process for generating a representation of a user, where the enrollment process includes capturing information about one or more physical characteristics of a user of the computer system.

Most Apple fans are used to an enrollment process that enables Face ID on their iPhones where you turn your head one way, then another and up and down and so forth.

In the last week, Patently Apple has covered two patents (01 and 02) regarding the enrollment process for Apple Vision Pro that goes one step further by creating a realistic avatar of the user that could be used in FaceTime calls.  

Today's patent application covers this process for Apple Vision Pro and for Apple Watch. 

Apple's patent FIG. 8 below is a flow diagram of methods of providing guidance to a user during a process for generating a representation of the user: FIGS. 7A,B,F and L illustrate example techniques of the enrollment process for generating a representation of a user and/or displaying the representation of the user on an Apple Watch.

You'll see in FIG.  7 music feedback when you've accomplished each stage of the enrollment. Considering that you can't see what you're doing, like with the enrollment process for Face ID on your iPhone, the triggered music will communicate with the user that an enrollment segment has been completed. 

Like with Vision Pro, the representation of the user is displayed and/or otherwise used on an Apple Watch to communicate during a real-time communication session (FaceTime call).

In some embodiments, the real-time communication session (FaceTime) includes displaying the representation of the user and/or outputting audio corresponding to utterances of the user in real time.

In some embodiments, the real-time communication session includes displaying the representation of the user (and, optionally, a representation of the second user) in an extended reality environment via display devices of the computer system and the second computer system.

Apple's patent FIGS. 7Q AND 7R below are more visuals of the enrollment process like for Face ID; in FIGS. 14C&D we see that the user is asked to enroll images of the user's hands so that they could be used with the realistic avatar when needed; FIG. 15 is a flow diagram of methods of prompting a user to position hands of the user in a plurality of poses.  

Apple's patent FIGS. 16 illustrates an example technique for adjusting an appearance of a representation of a user.

Apple's patent application is long and complicated. For lawyers, developers and geeks who want to dive into the finite details, review patent application 20230384860.