The U.S. Patent and Trademark Office officially published a series of 63 newly granted patents for Apple Inc. yesterday. In this particular report we briefly cover a series of four patents relating to Augmented Reality. And as always, we wrap up this week's granted patent report with our traditional listing of the remaining granted patents that were issued to Apple yesterday.

Method and System for representing a Virtual Object in a view of a Real Environment

Yesterday, the U.S. Patent Office granted Apple patent 11,182,961 which covers a method and system of representing a virtual object in a view of a real environment is provided which includes providing image information of a first image of at least part of a human face captured by a camera, providing at least one human face specific characteristic, determining an image area of the face in the first image as a face region, determining at least one first light falling on the face according to the face region and the at least one human face specific characteristic, and blending in the virtual object on a display device in the view of the real environment according to at least one first light.

The method further includes providing a plurality of face sample positions, providing for each of the plurality of face sample positions a radiance transfer function, determining image positions of the face sample positions in the first image, and determining a first light according to intensity information associated with the image positions of the face sample positions in the first image and the radiance transfer functions associated with the face sample positions in the first image.

Apple's patent FIG. 1 below illustrates a method of capturing a user's face in an image, and a general set-up; FIG. 4 shows a differential rendering step.

More specifically, Apple's patent FIG. 4 above illustrates a differential rendering step that can be used for blending in a virtual object on a display device in a view of the real environment according to the at least one first light determined in the illumination estimation step.

Based on a proxy object #401 which substitutes the 3D geometry and material part of the real-world scene that is influenced by the addition of the virtual content, two global illumination solutions are rendered. The first one is an image (#403) of the proxy object (#401) alone under the estimated lighting conditions (#402) of the real world.

The second one is an image #404 of the proxy object under the estimated lighting conditions #402 of the real-world with the illumination of the proxy object influenced by the virtual object, for example virtual glasses #405. The image #404 contains shadowing and light interreflections of the combined scene of proxy and virtual object, for example shadow #407 from the real scene as well as shadow #406 cast from the virtual object onto the real scene.

The first image #403 is subtracted (step #408) from the second image so that the resulting image #409 does only contain the difference in lighting between the two solutions like for example shadow regions #410 introduced by the virtual object, but not those shadow regions #411, that were already shadowed before.

To review the rich details of granted patent 11,182,961 click here.

Three Additional AR Patents Granted Yesterday

Granted Patent 11,182,964 titled "Tangibility visualization of virtual objects within a computer-generated reality environment." See Patently Apple's original patent report on this dated November 2019.

Granted Patent 11,182,978 titled "Rendering virtual content with coherent visual properties."

Granted Patent 11,182,980 titled "Procedural generation of computer objects."

Patently Apple also covered five additional granted patents yesterday that you could review here:

01: "Apple has Won a Patent for a Retractable MacBook Keyboard that could make Future MacBooks Thinner."

02: "Apple wins Patent for a Revolutionary next-gen Lenticular Display for a next-gen TV & Apple TV+ Content."

03: "Apple wins a Pair of Project Titan Patents covering Advanced Window & Suspension Systems."  

04: "Apple has won a patent for an HMD with an Adjustable Mechanism that will provide users with a customized facial fit.

05: "Apple won a pair of Smart Fabric patents today that may one day apply to smart clothing, wearables, vehicle fabrics & more."

Apple's Remaining Granted Patents for 11/23/2021