Today the US Patent & Trademark Office published a patent application from Apple that relates to generating three-dimensional geometric representations of physical environments, and in particular, to systems, methods, and devices that generate geometric representations based on depth information detected in physical environments.

Apple patents never present Apple's trademarks to describe matters. First comes the original patent filing, then comes the trademarks. While today's patent never mentions "Object Capture," the nature of what was described at this year's WWDC21 keynote regarding Object Capture is found in parts of today's published patent application.

Apple introduced "Object Capture" at WWDC21 back in June. Below is a quick overview of this "revolutionary API" for macOS. Object Capture uses Photogrammetry to turn a series of 2D images into photorealistic 3D Objects in just minutes.

The full WWDC21 Session titled "Create 3D models with Object Capture" could be viewed here. Apple has a ton of material including video on Object Capture for AR developers that could be reviewed here.

Apple's Patent application, titled "Single-Pass Object Scanning," covers devices, systems, and methods that generate a three-dimensional (3D) model using a selected subset of image data and depth data. The 3D model is generated based on images of a physical environment, depth information detected in the physical environment, and other information for tracking the devices/depth camera's particular position and orientation. It may be desirable to exclude images that include motion-based defects and/or include only a selected particular set of images (e.g., keyframes) for various reasons. Doing so may provide a more useful, realistic, or physically meaningful model of an object.

Some implementations of this disclosure involve an exemplary method of generating a 3D model of an object based on a selected subset of images and depth data corresponding to each of the images of the selected subset. The exemplary method initially involves acquiring sensor data during movement of the device in a physical environment including an object, the sensor data including images of a physical environment captured via a camera on the device.

For example, a user moves a device (e.g., an iPhone) around an object (e.g., a shoe or chair, as presented in the Object Capture video above) in a physical environment to capture images of the object from different sides. In some implementations, the sensor data may include depth data and motion sensor data.

In some implementations, the device includes a user interface, and during the movement of the device, the user interface displays the captured physical environment including the object (e.g., a live video stream), and a preliminary 3D model of the object based on the sensor data. In some implementations, the preliminary 3D model is generated during the movement of the device based on the selected subset of images, and the preliminary 3D model is displayed simultaneously with the images of the physical environment captured via the camera on the device (e.g., a picture-in-picture overlaid on the live video stream).

In some implementations, the generated 3D model of the object is based on generating a mesh, a 3D point cloud, or a voxel representation of the object.

In some implementations, the sensor data includes depth data (e.g., RGB-D) and light intensity image data (e.g., RGB) of the physical environment.

Apple's patent FIG. 7 below is a system flow diagram of an example generation of a 3D model based on a subset of images and depth data corresponding to each of the images of the subset information; FIG. 8 is a system flow diagram of an example generation of a live preview of a 3D model based on a subset of images and depth data corresponding to each of the images of the subset information.

Apple's patent application number 20210264664 could be best appreciated by Apple's many AR developers worldwide, though I'm sure that many super-geeks and 3D hobbyists will like to dive into the finer details of today's patent application as well.