Today the US Patent & Trademark Office published a patent application from Apple that relates to spatial audio. The patent delves into advancing sensors in devices from AirPods to AirPods Max to XR headset and more to determine the user's environment and adjust the audio accordingly. Apple notes that the more the virtual room resembles the acoustics of the real room in which the person is operating, the more plausible the sense of externalization will be.

Apple begins their patent application noting that content creators may create an audio or audio visual work. The audio may be fine-tuned precisely to the taste of the content creator in order to deliver a specific experience to a listener. The content creator may craft the audio so that it carries with it, perceivable cues of a particular scene, for example, an echoing outdoor mountainside, a stadium, or a small enclosed space. An audio work that is recorded outdoor may have perceivable acoustic cues that transports the listener to the outdoor environment. Similarly, if an audio work is recorded in a chamber, the listener may be virtually transported to the chamber.

A user may listen to an audio work in various location. Each location can have a different acoustic environment. For example, a user can listen to an audio or audio visual work in a car, on a grass field, in a classroom, on a train, or in the living room. Each acoustic environment surrounding a user may carry with it expectations of how sound is to be heard, even if the sound is being produced by headphones worn by a user.

Humans can estimate the location of a sound by analyzing the sounds at their two ears. This is known as binaural hearing and the human auditory system can estimate directions of sound using the way sound diffracts around and reflects off of our bodies and interacts with our pinna.

Microphones can sense sounds by converting changes in sound pressure to an electrical signal with an electro-acoustic transducer. The electrical signal can be digitized with an analog to digital converter (ADC). Audio can be rendered for playback with spatial filters so that the audio is perceived to have spatial qualities. The spatial filters can artificially impart spatial cues into the audio that resemble the diffractions, delays, and reflections that are naturally caused by our body geometry and pinna. The spatially filtered audio can be produced by a spatial audio reproduction system and output through headphones.

A spatial audio reproduction system with headphones can track a user’s head motion. Binaural filters can be selected based on the user’s head position, and continually updated as the head position changes. These filters are applied to audio to maintain the illusion that sound is coming from some desired location in space. These spatial binaural filters are known as Head Related Impulse Responses (HRIRs).

Spatial Audio - Virtual Room Presets

With Apple venturing into delivering realistic XR environments via their XR Headset, Apple has fine-tuned their spatial audio technology. According to Apple, a robust virtual acoustic simulation (e.g., spatial audio) benefits greatly from virtualization of a room to induce a sense of sound externalization which can be understood as the sensation of sound not coming from the headphones, but from the outside world. Deciding on the acoustic parameters of the virtual room is important to provide a convincing spatial audio experience.

Generally, the more the virtual room resembles the acoustics of the real room in which the person is operating, the more plausible the sense of externalization would be. However, when reproducing pre-recorded audio content such as a movie, a podcast, music, or other content, using spatial audio, emulating the real room can be detrimental to the experience because the acoustics of the virtual room may over-power or create a perceived discrepancy from the acoustics of the recorded content.

A typical example of this is an outdoor movie scene, in which a user may expect to hear no or little reverberation, but due to virtualization, the user may hear a significant amount of reverberation from the virtual room. In such cases, a trade-off or compromise can be made between reproduction plausibility (which aids in externalization and envelopment) and reproduction fidelity (which maintains the viewing experience as intended by the content creator).

In some aspects, a system may select an optimal virtual room preset or parameters of a reverberation algorithm based on analysis and/or a priori knowledge of the acoustics of the real room and the acoustics of the content being played-back.

Apple's patent FIG. 1 below illustrates audio processing of an audio or audiovisual file;  FIG. 2 illustrates selection of acoustic parameters for audio processing; FIG. 3 illustrates a workflow for determining and applying preset acoustic parameters.

Apple's patent FIG. 6 below illustrates audio processing operations for determining preset acoustic parameters; and lastly, FIG. 7 illustrates metadata with acoustic parameters.

The audio processing system can be a computing device such as, for example, a desktop computer, a tablet computer, a smartphone, a computer laptop, a smart speaker, a media player, a household appliance, a headphone set, a head mounted display (HMD), smart glasses, an infotainment system for an automobile or other vehicle, or other computing device.

For more details, review Apple's patent application number US 20230104111.