A little over a year ago, Apple's CEO Tim Cook characterized Augmented Reality (AR) technology as "The Next Big Thing." Beyond gaming, Cook touched on the practical side of life with AR by noting that there will be 'how-to' applications to help with practical hands-on tasks. Cook noted that "You may be under the car changing the oil, and you’re not sure exactly how to do it. You can use AR, to explain it all."

Today the US Patent & Trademark Office published a patent application from Apple that relates to synthesized reality (SR or AR) content consumption, and in particular, to systems, methods, and devices for tailoring an SR (AR) experience to a physical setting. Some examples provided describe how entertainment on something like Apple TV+ could take a huge leap in the decade ahead.

Apple's patent abstract places a lot of attention on method of creating future AR based on Mesh mapping. Apple notes that "In one implementation, a method includes obtaining locality data characterizing objects and relative spatial information of a volumetric region around a user; synthesizing a mesh map of the volumetric region based on the locality data; selecting synthesized reality (SR) content based on the mesh map, wherein the SR content satisfies a dimensional variance threshold relative to one or more portions of the mesh map; compositing at least a portion of the SR content with the mesh map in order to generate composite SR content; and presenting the composite SR content to the user in order to occlude at least a portion of a visual presentation of the volumetric region. In some implementations, the SR content is adapted to fit the one or more portions of the mesh map. In some implementations, the SR content is updated as the user location changes or the user interacts with the SR content.

Apple further details an electronic system that may have an opaque display and at least one imaging sensor for capturing images or video of the physical setting, which are representations of the physical setting. The system combines the images or video with virtual objects, and displays the combination on the opaque display.

An individual using the system, views the physical setting indirectly via the images or video of the physical setting, and observes the virtual objects superimposed over the physical setting.

When a system uses image sensor(s) to capture images of the physical setting, and presents the AR setting on the opaque display using those images, the displayed images are called a video pass-through.

Alternatively, an electronic system for displaying an AR setting may have a transparent or semi-transparent display through which an individual may view the physical setting directly. The system may display virtual objects on the transparent or semi-transparent display, so that an individual, using the system, observes the virtual objects superimposed over the physical setting.

Apple's patent FIGS. 7A-7B below illustrate an example SR presentation scenario. In one you could see how there could be store layout and no products and with AR, the user will be able to see a store full of merchandise.

Yet Apple's example goes miles deeper than you'd expect. Apple notes that the HMD user will use a controller or an iPhone to detect a command issued by user to enter an SR experience associated with the video content.

In response to detecting the command, for example, the controller synthesizes a mesh map of the physical setting (#105) and detects planes within the mesh map.

Those wanting to deeper down the rabbit hole regarding the mesh map could read the details of this scenario beginning at patent point #0091 through to #0094.

The Next Great Thing: AR Entertainment 

Then Apple dives into a series of possible future scenarios to illustrate how real AR is going to get in the future.

In one example, Apple notes that video content viewed on the HMD could correspond to a court room scene within a movie. Apple describes how users will be able to see the court room with lawyers, judge and all playing out as if it's all in the user's living room.

Another example describes how sports fans will be able to view a boxing match, viewing fighters sparring within a boxing ring with the crowd in the background yet seem as if the entire fight is within the user's living room.

Apple is providing us with a view of the future of AR that could play out on Apple TV+ content running on mixed reality headsets and it certainly sounds wild. Let's see how Netflix competes with that.

Apple's patent FIG. 8 below illustrates a flowchart representation of a method of tailoring an SR experience to a physical setting.

This is a lengthy patent filing covering a lot of territory. You could review the finer details by reviewing Apple's patent application number 20210056749.

Considering that this is a patent application, the timing of such a product to market is unknown at this time.