Today the US Patent & Trademark Office published a patent application from Apple that relates to methods and systems for generating three-dimensional models of a user's face in captured images. A dimension of this invention was likely used to create the Messages Memoji app. The majority of the inventors came from Faceshift that Apple acquired back in 2015.

Apple notes that motion capture has been used in a variety of areas to generate motion data that is based on tracking and recording the movements of real objects. For example, motion capture technology has been used frequently in video game production and movie production.

Motion capture technology, however, has not been widely implemented at the consumer level. Consumer level motion capture systems have just begun to be implemented as processing and power advancements begin to allow consumer based electronics to more readily perform operations associated with motion capture.

Apple's invention covers a three-dimensional model (e.g., motion capture model) of a user that is generated from captured images or captured video of the user. A machine learning network is used to track poses and expressions of the user to generate the three-dimensional model from the capture images.

The machine learning network may refine the three-dimensional model to provide a more accurate tracking of the user's face. Refining of the three-dimensional model may include defining selected locations in the model (e.g., eye corners, nose, etc.) and refining the determinations of poses and expressions based on the three-dimensional model being projected onto the captured images. The three-dimensional model may then be refined using the refined poses and expressions. The refining may occur in an iterative process. Tracking of the three-dimensional model over time (e.g., during video capture) may be used to generate an animated three-dimensional model of the user that simulates the user's poses and expressions.

While this technology could be used for Face ID, the motion aspect shows us that this is likely one of the initial patents supporting the creation of a user's Memoji.

Apple's patent FIG. 5 below depicts a representation of a model of a user's face; FIG. 6 depicts a side-by-side representation of an example captured image and an example three-dimensional model projected onto the example captured image; FIG. 7 depicts an example of two three-dimensional models of a user with different poses and expressions in each of the models.

In respect to FIG. 5, Apple notes that in some embodiments, model #300 is a "blendshape" model of the user's face. The model includes a selected number of muscle sets (e.g., blendshapes). The muscle sets may be defined for different muscle movements of the user's face such as, but not limited to, eyebrow movement up or down, cheek squint, chin lower or raising, eye blink, eye movement up or down, eye movement in or out, eye open or closed, eye squint, jaw left or right, jaw open or closed, lips opened or closed, lip pucker, lip stretch, mouth frown, and mouth smile. Movement (e.g., deformation) of these muscle sets may individually, or in combination, produce local deformations of the user's face. Thus, localized movements (deformations) of the user's face and the expression for the user's face in the image may be determined by assessing the deformations of the muscle sets in the captured image.

Apple's patent FIG. 4 below depicts a flowchart of an embodiment of a process to generate a three-dimensional model from a captured image.

Apple's patent application 20190180084 that was published today by the U.S. Patent Office was originally filed back in Q1 2018. Considering that this invention was likely used in creating Emoji in Messages, this becomes a patent fulfilled.

Some of the Inventors

Thibaut Weise: Software Engineering Manager. Weise was the CEO and Co-founder of Faceshift.

Brian Amberg: Software Engineering Manager: Former CTO and Co-Founder of Faceshift

Stefan Brugger: Senior Computer Vision Engineer. Came to Apple via Faceshift

Patrick Snape: Machine Learning Engineer (Zürich Area, Switzerland).

About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.