Today the U.S. Patent and Trademark Office officially granted Apple a patent that relates to audio systems used to render spatial audio in context with Augmented Reality. In one case, the patent covers spatial audio used in a head mounted display device (HMD) and one that could integrate an iPhone as its display.

Headphones typically play music by rendering a stereo audio image that is perceived by the listener as internalized sound coming from within the listener's head. Headphones can also be used to render spatial audio to create a more natural experience, e.g., to simulate externalized sound coming from a location outside of the listener's head. An effective spatial sound reproduction is perceived just as the user would experience the sounds in the real world. Creating such a spatial sound reproduction, however, is a challenging and ongoing endeavor.

Several components contribute to an effective spatial sound rendition that creates the illusion of an externalized virtual sound source. First, an accurate head-related transfer function (HRTF) personalized to the listener contributes to effective spatial audio. An HRTF is a model of a filter that contains all of the acoustic information required to describe how sound reflects or diffracts around a listener's head, torso, and outer ear before entering their auditory system. The HRTF matches the reproduced audio to the user anatomy to make the audio sound as if it is actually interacting with the user anatomy along an acoustic path from the external location to an ear entrance.

Second, an environment-related transfer function (ERTF) specific to a listening environment contributes to effective spatial audio. An ERTF is a model of a filter containing acoustic information required to describe how sound reflects and reverberates within the listening environment. The ERTF matches the reproduced audio to the listening environment to make the audio sound as if it is actually coming from within the listening environment.

Third, head tracking of the listener's head during audio reproduction contributes to effective spatial audio. Head tracking involves tracking a position and/or orientation of the listener's head and using the tracking information to shift the virtual sound source accordingly. Shifting the virtual sound source opposite to movement of the listener's head can cause the listener to perceive virtual sound as coming from a fixed location within the listening environment as the user moves his head or body. In scenarios where personalized HRTFs and ERTFs cannot be obtained, however, a system can use other representations to provide visual cues to a user regarding the presence of virtual sound sources.

In an aspect, an audio system provides a visual reference corresponding to a virtual sound source. For example, the audio system can receive a user input on a display of a device to designate one or more positions within an augmented reality view of a listening environment. The positions can correspond to one or more sound source locations within the listening environment.

A corresponding sound source icon, e.g., an image of a loudspeaker, can be displayed at the position on the display to be viewed by a listener.

The audio system can reproduce a binaural audio signal, e.g., using a headset, to render spatial audio including a sound coming from the sound source location within the listening environment. The listener can listen to the spatial audio, which may be music or a movie soundtrack, even after setting down or turning off the display of the device.

The experience of creating and/or viewing the sound source icon on the display before or during the spatial audio rendition can cause a durable spatial audio experience that is less likely to collapse into the head of the listener.

Apple's patent FIG. 1 below represents a flowchart of a method of augmenting spatial audio rendition; FIG. 2 is a pictorial view of a user handling an audio system.

The user can perform a set up process to place one or more sound source icons, e.g., visual representations of speakers, at specified positions in an augmented reality view on a display (#204).

In an aspect, the user can open an audio application executing on mobile device. The audio application can be a music application or a movie application that renders spatial audio to the user. More particularly, the spatial audio rendition can be music or a movie soundtrack. The user can perform operations within the audio application to initiate an augmented reality view.

The augmented reality view can include the presentation of one or more images (#202) of the listening environment (#210). For example, the one or more images of the listening environment can be frames of a video of listening environment. The listening environment can be the actual environment in which audio system is being used by the user, e.g., a car, a living room, etc. Alternatively, the listening environment can be a simulated space that the user wishes to emulate in the spatial audio rendition, e.g., a concert hall, a movie theater, etc. In any case, image(s) of the listening environment can be presented to the user on display.

Apple's patent FIG. 3 below is a pictorial view of a sound source icon overlaid on one or more images of an actual listening environment; FIG. 4 is a pictorial view of a sound source icon overlaid on one or more images of a simulated listening environment. the user can enter a command within the audio application, e.g., tap on a listening environment option that is presented on display and titled "Famous Movie Theater," to enter the augmented (or virtual) reality view in which an augmented (or virtual) reality scene of a predetermined well-known movie theater is presented on display (FIG. 4).

The user can view images of the listening environment to find positions (#208) in the augmented reality view where sound sources are to be placed.

Lastly, Apple notes that in one aspect, the audio system can be a head-mounted display (HMD). The HMD can integrate both the device (#212) and the headset (#215) into a single device. The HMD can be a display device that incorporates the display used to present the visual representation of the sound source at the location, and the headset used to render spatialized audio to localize sounds to the locations. The HMD can be worn on a head of the user to allow the user to see images presented on the display and hear sounds played by the headset.  

The HMD using the iPhone's display supports a long-standing invention of Apple's. In fact, on August 10, Patently Apple posted a report covering Apple's 15^th granted patent for this invention as presented below. It goes to prove that Apple is indeed continuing to work on this entry-level HMD.

For more details, review Apple's granted patent 11,102,578.