Yesterday Patently Apple posted a report titled "An Apple patent reveals an AR Interface feature for Headsets & Glasses that will allow users to control external devices with in-air gestures." The patent related to AR interfaces, and more specifically, to techniques for controlling external devices like a TV using a next generation augmented reality interface included with mixed reality headsets and glasses that work with in-air gestures as noted in our report's cover graphic. One of the key inventors of this patent was Michael Kuhn, Apple's software development manager that came to Apple via the acquisition of Germany's Metaio in 2015.

On Thursday, the US Patent & Trademark Office published a second patent application from Apple that originally came from Metaio, this time with the one-time owner of Metaio Peter Georg Meier, who is now Apple's Director, Algorithm Lead, Technology Development Group. One of the reasons for acquiring Metaio was for their advanced work in Virtual Reality (AR/VR/MR) and hologram technologies.

Hologram technology is touched on in this patent. In patent point #0024 it's noted that "a system may have a projection system that projects virtual objects into the physical environment, for example, as a hologram or on a physical surface, so that a person, using the system, perceives the virtual objects superimposed over the physical environment." Think of a virtual TV projected on any wall in your home. More on that later.

One of Metaio's great inventions that Apple has acquired relates to "Thermal Touch" which is a vision of the near future for wearable computing user interfaces. By fusing information from an infrared and standard camera, nearly any surface can be transformed into a touch screen.

In some ways, Apple's patent application titled "Scene Classification," invented by Peter Meier, touches on Thermal Touch technology. In patent point #0039 Apple notes: "In some embodiments, the system includes touch-sensitive surface(s) for receiving user inputs, such as tap inputs and swipe inputs. In some examples, display(s) and touch-sensitive surface(s) form touch-sensitive display(s)."

Whether Apple will rebrand this technology when their MR Headset arrives is unknown at this time.

Overall, Apple's patent application covers techniques for providing content using a computer-generated reality interface depicting virtual objects in combination with a representation of a physical environment.

In one exemplary technique, image data corresponding to the physical environment are obtained using one or more cameras. At least one portion of an entity in the physical environment is identified based on the image data. Based on the identified at least one portion of the entity, whether the entity is an entity of a first type is determined. One or more virtual objects and a representation of the entity are displayed.

The system may be configured to present virtual objects on the transparent or translucent display, so that a person, using the system, perceives the virtual objects superimposed over the physical environment.

Alternatively, a system may have an opaque display and one or more imaging sensors that capture images or video of the physical environment, which are representations of the physical environment.

The system composites the images or video with virtual objects, and presents the composition on the opaque display. A person, using the system, indirectly views the physical environment by way of the images or video of the physical environment, and perceives the virtual objects superimposed over the physical environment.

Apple's patent FIG. 2A below depicts a user device displaying a representation of an indoor physical environment; FIG. 2B depicts a block diagram of a user device including classifiers configured to identify one or more entities of an indoor physical environment.

Apple's patent FIGS. 2C and 2D above depict various flows for classifying identified entities and determining the type of the classified entity. Here we're able to see that everything in that photo scene taken by an iPhone with its TrueDepth camera can add AR information about the sink, the cat, the appliances and so forth.

Apple's patent FIG. 2H below depicts a user device displaying a computer-generated reality interface including virtual objects and representations of identified entities.

Each room of the house can be set to an augmented reality world. When in the kitchen, the user will be able to ask Siri to call up a recipe to follow and it will pop-up in the users view inside a head-mounted device (HMD).

In the living room, the HMD could correspond to virtual objects, such as controls for operating a TV, a user-interaction mechanism for ordering movies, or a user-interaction mechanism for subscribing magazines.

Further, the headset could work with an augmented reality TV guide service or TV subscription service can be provided regardless of whether the TV entity is located in a bedroom or a living room.

Accordingly, the user device is configured to present one or more virtual objects based on the identified TV entity (e.g., a virtual object enabling the user to receive an on-line movie streaming service) without having to determine the type of physical environment (e.g., whether the physical environment is a living room or bedroom).

Other scenarios are discussed in the patent filing, such as one relating to outdoor environments. Apple's patent FIG. 3E below depicts a user device displaying a computer-generated reality interface including virtual objects and representations of identified entities.

A user may be walking in a park when a message could pop up to notify the user that a concert is coming to the park based on GPS sensors and enable the user to order tickets for a particular concert in the park.

Apple's patent application number 20210365686 continues to advance this invention by adding 24 new patent claims to better protect this invention from competitors and patent trolls. You could review the patent in full here.

Th patent highlighted today along with the one that we reported on yesterday provides a glimpse at some of the AR interface attributes that could be coming our way in the not-too-distant future.  

Patent #2: Augmented Visual Capabilities

The U.S. Patent Office also published AR patent application 20210364809 from Apple on Thursday titled "Augmented Visual Capabilities." You could review it here. The invention carries a sci-fi quality to it that could be interesting in the future.  

About FIG. 8 below: As an example, occlusion object #710 may represent an exterior wall of a night club, the obstructed area may represent a dance floor of the night club, and object #720 may represent one or more patrons of the night club. In this example, a user of the device standing outside of the night club may wish to see how many patrons are in the night club before entering. Continuing with this example, to request enriched view #800 of FIG. 8 (and thereby see how many patrons are in the night club before entering), the user may interact with input device (HMD) in a manner indicative of the request. In accordance with detecting such an interaction with HMD, an enrichment effect is applied to a subset of the image data corresponding to the obstructed area to form content representing enriched view #800 of FIG. 8.