Today the US Patent & Trademark Office published a series of three patent applications from Apple that relate to next-gen AR/VR/MR and Extended Reality Environments in one way or another.

Patent #1: Environmental Application Model

Various devices are used to provide users with XR environments. The content of such XR environments may be experienced by a user using a mobile device, head-mounted device ("HMD"), or other device that presents the visual or audio features of the content. The experience can be, but need not be, immersive, e.g., providing most or all of the visual or audio content experienced by the user. The content can include pass-through video (e.g., in which real-world content is captured by a camera and displayed on a display with additional content) or optical-see-through (e.g., in which real-world content is viewed directly or through glass and supplemented with displayed additional content).

In some instances, the content can also include visual representations of multiple applications. Some devices and software programs facilitate the display of such content that includes visual representations of multiple applications. However, these devices and software programs do not provide sufficient features for resolving any conflicts arising among the multiple applications concerning views of XR environments.

Apple's invention covers various implementations that relate to techniques for customizing an ambience of a XR environment presented to a user. In one implementation, a method includes presenting content representing a view of an XR environment at an electronic device on a display of the electronic device. The XR environment comprising visual representations corresponding to a plurality of applications.

A first application among the plurality of applications is designated as an elevated application. The elevated application is provided with access to a control parameter configured to modify an ambience of the XR environment.

Other applications of the plurality of applications are precluded from accessing the control parameter while the first application is designated as the elevated application.

While Apple's patent mainly applies to a Head Mounted Device (HMD), Apple notes that their invention could apply to other devices in the future including heads-up display, vehicle windshield, projection systems, contact lenses, smartglasses, an iPhone, iPad or MacBook.  In one of more applications a controller may be used to control XR environments, which could be hinting at games, though not limited to games.  

The patent describes a user with wearing an HMD running multiple applications virtual checker while a weather app shows whether conditions quite extensively in the HMD environment.  

In Apple's patent FIG. 7 below we see a user input received via the weather application may change that value of the control parameter to a value that defines a second state in which that portion of the XR environment is populated with computer-generated content. In one implementation, the computer-generated content corresponds to a functionality provided by the elevated application. Responsive to detecting that change to the value of the control parameter, the XR creation unit generates computer-generated whether related content (#712, 714, and 716.)

In one implementation, user input received via a particular application among multiple applications concurrently designated as an active application and an elevated application may modify both a local ambience associated with the particular application and an environmental ambience.

(Click on image to Enlarge)

For more details on Apple's patent application 20210279966 titled "Environment Application Model" click here. One of the lead inventors on this patent filing is Alexis Palangie, a Senior Software Engineer who previously worked at Ubisoft, Lucas Arts and Oculus VR.   

Patent #2: Object Centric Scanning

Two weeks ago, Patently Apple posted an IP report titled "A Patent Application Published Today appears to cover Apple's 'Object Capture' that was Introduced at WWDC21." Today, another patent on this same subject has come to light titled "Object Centric Scanning."

Apple's patent covers devices, systems, and methods that generate a three-dimensional (3D) model of an object based on images and positions of a device during acquisition of the images. The 3D model is generated based on images of a physical environment, depth information detected in the physical environment, and other information for tracking the devices/depth camera's particular position and orientation (e.g., camera pose). It may be desirable to utilize a camera pose that is object centric, e.g., defined in terms of an object-based coordinate system. In other words, constrained poses are tied to a target object (e.g., a single object of interest), and position is relative to the target object. Doing so may provide a more useful, realistic, or physically meaningful model of an object.

Apple's patent FIG. 4 below is a flowchart representation of an exemplary method that generates a three-dimensional (3D) model of an object based on images and tracked positions of the device during acquisition of the images; FIG. 5A is a block diagram illustrating example positions of a camera with respect to an object in a physical environment; and FIG. 9 is a system flow diagram of an example generation of a live preview of a 3D model of an object based on images and tracked positions of the device during acquisition of the images. 

For details, review Apple's patent application 20210279967. Apple's inventors include:

Thorsten Gernoth: Computer Vision Engineering Manager

Chen (Kimi) Huang: Senior Computer Vision / Deep Learning Research Engineer

Shuo (Steven) Feng: Computer Vision Engineer

Hao Tang: Productizing computer vision and machine learning

Tobias Rick: Video Engineering. Previously worked at Samsung as a Technical Lead Software Engineers – Next Experience Display Lab

Patent #3: Object Detection using Multiple 3D Scans

Patently Apple covered this invention in a granted patent report posted on August 24, 2021. Two weeks later and the U.S. Patent Office has published Apple's update to their granted patent. Apple has added 20 new patent claims to better protect their invention that you could review in full in their continuation patent #20210279898 here.