Back in 2015 Patently Apple reported on a rumor that Apple had quietly acquired a real-time motion-capture company by the name of 'Faceshift.' The technology was likely later used to create Animoji and Memoji. Then in 2019, Apple acquired IKinema, an amazing company that produced technology that mimicked full body avatar movements and detailed facial expressions. Below is a 12-minute TED Talk video about their technology in action. It's amazing stuff.

This past summer Apple was granted patent 11,062,476 that was driven by two engineers from Apple's acquisition of Metaio. In that patent's background they stated that "head-mountable devices (HMDs) present immersive CGR experiences to a user of the HMD. Some CGR experiences require knowing a body pose of the user. For example, some CGR experiences present an avatar of the user that mimics the behavior of the user. If the user moves a portion of their body, the avatar moves the corresponding portion." This is totally inline with IKinema's technology.

Yesterday the US Patent & Trademark Office published a patent application from Apple titled "Presenting Avatars in Three-Dimensional Environments." One part of Apple's patent covers hand tracking / gesture tracking, a feature that Ming-Chi Kuo emphasized yesterday in a new report regarding Apple's second generation headset. A second aspect of the patent covers eye-tracking.

Where the patent gets really interesting begins at patent FIG. 7A that describes filming a user (actor) on an animated movie set.

Apple's patent FIG. 7A below depicts a user (#701) standing in real environment (#700) with both arms raised and the user's left hand holding a cup (#702). In some embodiments, the real environment is a motion capture studio that includes cameras 705-1, 705-2, 705-3, and 705-4 for capturing data (e.g., image data and/or depth data) that can be used to determine a pose of one or more portions of the user.

The poses of the portions of a user are used to determine a pose of a corresponding avatar character (e.g., see avatar 721 in FIG. 7C further below) in a CGR environment such as, for example, an animated movie set.

In Apple's patent FIGS. 7B and 7C above, Apple points to a "certainty map" (#710) which is a visual representation of the computer system's certainty of the determined poses for portions of a user. In other words, a certainty map represents calculated estimations of the certainty of the positions of different portions of the user's body that are determined using the computer system (e.g., probabilities that the estimated positions of the portions of the user's body are correct).

A certainty map is a template of a human body that represents basic human features such as a head, neck, shoulders, torso, arms, hands, legs, and feet.

In some embodiments, a certainty map represents various sub-features such as fingers, elbows, knees, eyes, a nose, ears, and a mouth. The human features of a certainty map correspond to the physical features of a user.

Apple's patent FIG. 10A depicts an example in which a user is represented in a CGR environment as virtual avatar characters that have an appearance based on different appearance templates.

In some embodiments, the avatars have an appearance that is based on a character template or an abstract template. In some embodiments, the avatar changes appearance by changing poses. In some embodiments, the avatars change appearance by transitioning between a character template and an abstract template.

(Click on image to Enlarge)

Apple's patent FIG. 12 below is a flowchart of an exemplary method (#1200) for presenting avatar characters that have an appearance based on different appearance templates.

(Click on image to Enlarge)

The method (#1200) could be performed on an iPhone, iPad, MR Headset that is communication with a display generation component. In some embodiments the method could apply to a visual output device, a 3D display, a transparent display, a projector, a heads-up display, a display controller or touchscreen.

Apple further notes that a template of an animated character could be that of a human, a cartoon character or an anthropomorphic construct of a non-human character such as a dog, a robot, etc.

While it's very clear that Apple could use this technology for future Apple TV+ productions as outlined in this patent, just how it'll apply for Apple devices isn't exactly clear. Will future iPhones and iPads provide multiple TrueDepth Cameras? Will Apple sell specialty standalone depth cameras for an in-home studio? While only time will tell, it's a fascinating patent overall that will one day make it to market.

To review the depths of this invention, review Apple's patent application number 20210382544.  

Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Apple Inventors

Nic Henderson: Apple Design Team (In Henderson's LinkedIn Profile he mentioned that the design team was featured in the recent Wallpaper article titled "Inside Apple Park: first look at the design team shaping the future of tech."

(Click on image to Enlarge)

Giancarlo Yerkes: Human Interface Designer; Art Direction & Motion Design

Jason Rickwald: Designer, Human Interface

Rupert Burton: Human Interface Designer

Nico Scapel: Designer - Human Interface

Dorian Dargan: Producer, Human Interface Design

Gary Ian Butcher: No LinkedIn profile found. Butcher has been listed on many Apple patents over the years. Here are just two of them in 2021: 01: "Multi-dimensional Object Rearrangement," (for Apple Watch) and 02: "Breathing Sequence User Interface" (for Apple Watch).