Patently Apple posted a granted patent report in September 2019 that covered three inventions with the most import one focused on augmented reality. One of the inventors is noted as Sebastian Knorr from Germany who stayed on with Apple when they acquired Metaio back in 2015. Months later Apple introduced ARKit with some industry insider saying that developer tool came by way of Metaio's SDK which allowed Apple to jump into AR at a much faster pace.

Apple's 2029 granted patent FIG. 7 below illustrates a user experiencing an augmented reality scenario on a display device such as a tablet showing an image of a real scene--for example a table #708--in front of the user combined with virtual content--for example a rabbit #712.

The augmented reality example from the patent with Metaio's technology was introduced to developers in June 2017 with Apple's first major AR demo played out on stage by Wingnut. The demo was set on a table on stage using an iPad. The Augmented Reality app was able to place an entire AR world scene on the table. It was an amazing to see it unfold.  

Apple's patent FIG. 13 below shows a flow diagram of an embodiment for representing a virtual object in a view of a real environment.

While the patent behind this technology was granted to Apple in September 2019, yesterday the US Patent & Trademark Office published a continuation patent showing that they wanted to legally protect more of this invention.

Technically speaking, a continuation patent isn't a new patent but rather a patent wherein Apple is updating and/or adding specific technology, features and/or methods that they want to legally extend into their current granted patent. All of the changes or additions found in yesterday's continuation patent are restricted to Apple's patent claims.

At times reviewing patent claims can reveal some very interesting and key new dimensions of an invention. Yesterday's continuation patent does add a number of patent claims that could be important for legal professionals and perhaps developers who understand the nature of Apple's invention and can decipher the importance of the new claims. For the rest of us, it's a little difficult to clearly understand the value of the new patent claims that Apple legal wants to specifically protect.   

One of the key aspects of the patent was creating a 3D facial model using a multi-camera on a device with one of the cameras being a depth camera. Technically the technology is in concert with realtime facial tracking which later became Apple's Animoji.

The claims of the 2019 granted patent focused on a system and "A method of representing a virtual object in a view of a real environment, comprising: obtaining image information for a first image of at least part of a human face in a real environment captured by a first camera; determining at least part of an image area of the face in the first image as a face region of the first image; determining a first light source comprising: obtaining at least part of a 3D face model," and so forth.

In yesterday's continuation, Apple shakes up the order of the claims putting the emphasis of "the non-transitory computer readable medium. Apple adds a new claim on this front which covers a "radiance transfer function."

Apple's patent claims #6 states the following: "…wherein the instructions to determine the first light source comprises instructions to: obtain image information of a plurality of images of faces in a plurality of different illuminations and poses, wherein the plurality of different illuminations and poses are known, provide a plurality of face sample positions, determine, for each of the plurality of face sample positions, a radiance transfer function according to intensity information associated with at least part of the plurality of images and the illuminations and poses associated with the at least part of the plurality of images, and determine the first light source according to intensity information associated with the face sample positions in the first image and the radiance transfer functions."

Apple adds one the new claim (#13) relating to "The System" which notes the following: "under The system of claim 8, wherein the computer code to determine the first light source comprises computer code to: obtain image information of a plurality of images of faces in a plurality of different illuminations and poses, wherein the plurality of different illuminations and poses are known, provide a plurality of face sample positions, determine, for each of the plurality of face sample positions, a radiance transfer function according to intensity information associated with at least part of the plurality of images and the illuminations and poses associated with the at least part of the plurality of images, and determine the first light source according to intensity information associated with the face sample positions in the first image and the radiance transfer functions."

New Claims were added which relate to The Method of the system. Claim #15 states: "A method for generating a virtual object, comprising: obtaining image information for a first image of at least part of a face captured by a first camera; determining a face region in the first image as a face region of the first image; determining a first light source according to intensity information associated with the face region, a pose of the human face, and at least part of a 3D face model; and generating a virtual object with an illumination according to the determined first light source."

The second additional claim Apple made to "The Method" is Claim #19 which states the following: "The method of claim 15, wherein determining the first light source comprises: obtaining image information of a plurality of images of faces in a plurality of different illuminations and poses, wherein the plurality of different illuminations and poses are known, providing a plurality of face sample positions, determining, for each of the plurality of face sample positions, a radiance transfer function according to intensity information associated with at least part of the plurality of images and the illuminations and poses associated with the at least part of the plurality of images, and determining the first light source according to intensity information associated with the face sample positions in the first image and the radiance transfer functions."

Apple's continuation patent application 20200013223 that was published yesterday by the U.S. Patent Office was filed back on September 16, 2019 or one day prior to their first granted patent being published.

For developers, legal professionals or super geeks who want to dig deeper, you could review the patent claims of the September 2019 granted patent claims here with yesterday's continuous patent claims here to confirm the additions and a couple I may have missed or ignored.  

Predictive, Foveated Virtual Reality System

Another continuation patent about a future eye-tracking system invention related to a headset was published yesterday. Patently Apple first covered this in a granted patent report back in June 25, 2019 titled "Apple wins a Patent for a Predictive, Foveated VR Headset."

Apple's granted patent covered methods and systems for a virtual reality (VR) and/or augmented reality (AR) device (e.g., a headset, or head mounted, device) may include a predictive, foveated virtual reality system. A predictive, foveated virtual reality system may be configured to capture views of the world around a user of the system, augment the captured data, generate an augmented view of the world and display that view to the user via a display of the system.

Apple's patent FIG. 1 below is a logical diagram illustrating part of a system configured to implement a Predictive, Foveated Virtual Reality System; FIG. 4 and FIG. 5 are a logical diagrams illustrating one embodiment of Predictive, Foveated Virtual Reality System configured to capture image data centered on multiple different angles of view.

Yesterday the U.S. patent Office published Apple's continuation patent 20200012106 for a key eye tracking system. Unfortunately the changes were finite and I was unable to find any clear additions made to Apple's patent claims.

Did Apple legal screw up their continuation patent filing by filing the very same claims as their granted patent? Only time will tell. You could check the full patent here.