Today the U.S. Patent and Trademark Office officially published a series of 25 newly granted patents for Apple Inc. In this particular report we briefly touch on a series of patents relating to AR/VR/MR/XR development processes that developers my find interesting. Apple was also granted a design patent for an icon. And as always, we wrap up this week's granted patent report with our traditional listing of the remaining granted patents that were issued to Apple this week.

Excluding Objects during 3D Model Generation

Apple's patent background notes that physical environments have been modeled (e.g., reconstructed) by generating three-dimensional (3D) meshes, utilizing 3D point clouds, and by other means. The reconstructed meshes represent 3D surface points and other surface characteristics of the physical environments' floors, walls, and other objects. Such reconstructions may be generated based on images and depth measurements of the physical environments, e.g., using RGB cameras and depth sensors.

Existing techniques for generating 3D models based on images of a physical environment and depth information detected in the physical environment may be inaccurate and inefficient when there are moving objects (e.g., people, animals, etc.) in the captured image information. Additionally, when reconstructing a physical environment, it may not be desirable to include certain objects in the 3D model. It may be desirable to exclude objects that are moving or that are considered “movable objects.” For example, when an object is moving while light intensity information is captured by an RGB camera, and depth information is captured by a depth camera, there may be distortion and noise created during image acquisition. Accordingly, the noise generated from a moving object may result in undesirable or unintended appearances, for example, resulting in 3D representations that are inaccurately displayed and other noticeable issues.

Apple's invention covers various implementations that include devices, systems, and methods that generate a three-dimensional (3D) model that excludes a person or other object that is likely to move using depth data and an object mask generated from RGB image data. The 3D model is generated based on images of a physical environment, depth information detected in the physical environment, and other information for tracking the devices/depth camera's particular position and orientation. It may be desirable to exclude various objects from the 3D model that is generated for various reasons. For example, it may be desirable to exclude objects that are not typically fixed parts of a physical environment (e.g., people, animals, etc.). Doing so may provide a more useful, realistic, or physically meaningful model. As a specific example, it may be desirable to model a physical environment in which a person is sitting on a chair by modeling the chair without the person. 

Apple's patent FIG. 6 below is a system flow diagram of an example generation of a 3D model using 3D data and a segmentation mask based on depth and light intensity image information.

Click on image to Greatly Enlarge)

Developers could dive deeper into Apple's granted patent 11640692 here.

Other Granted Patents relating to AR/VR/XR Environments

01: Scene graph-based scene re-identification

02: Generating A Volumetric Representation Of A Capture Region

 Key Design Patents

Apple was granted a single design patent today under number D985020 that's titled "Display screen or portion thereof with icon." I must have missed the memo on this one. If someone happens to know what this icon represents, please send us a note in the comment section below or on our Twitter feed.

Today’s Remaining Granted Patents

Last Thursday we noted in one of our reports that it was unknown at this time if the drastic drop in Apple's patent applications and granted patents over the last two months is reflecting an issue with the U.S. Patent Office or reflecting the effect of COVID-19 had on Apple's ability to process patents during those years. Today, for instance, Apple was only granted 25 patents in contrast to 50-100 in the past. That's a dramatic drop. With that said, below is a list of the remaining granted patents issued to Apple today.