Patently Apple has covered a few "Binaural Audio" patents in the last year (01 and 02). We've also covered "Spatial Audio" in a patent report here as well as covering Apple's WWDC20 event where Spatial Audio was highlighted as coming to their next-gen AirPods Pro. Today the US Patent & Trademark Office published a patent application from Apple that relates dives into Binaural Audio for AirPods Pro and their next-gen Head Mounted Displays Device (HMD). More specifically, this patent introduces us to predictive head-tracking Binaural Audio. This aspect of audio could be a game changer for VR RPG gaming, horror movies, action moves and more where it could put you right in the middle of the action that the characters are in.

In Apple's patent background they note that virtual reality (VR) allows users to experience and/or interact with an immersive artificial environment, such that the user feels as if they were physically in that environment. For example, virtual reality systems may display stereoscopic scenes to users in order to create an illusion of depth, and a computer may adjust the scene content in real-time to provide the illusion of the user moving within the scene.

When the user views images through a virtual reality system, the user may therefore feel as if they are moving within the scenes from a first-person point of view. Similarly, mixed reality (MR) combines computer generated information (referred to as virtual content) with real world images or a real-world view to augment, or add content to, a user's view of the world, or alternatively combines virtual representations of real-world objects with views of a three-dimensional (3D) virtual world.

The simulated environments of virtual reality and/or the mixed environments of mixed reality may therefore be utilized to provide an interactive user experience for multiple applications.

In order to make these VR environments as realistic as possible, Apple has designed head-tracking binaural audio rendering in addition to spatial audio that could be a game changer for music, high-end gaming and movies.

Apple's invention covers various embodiments of methods and apparatus for predictive head-tracked binaural audio rendering. Embodiments of an audio rendering system and audio rendering methods are described that may, for example, be implemented by mobile multipurpose devices such as smartphones, pad devices, and tablet devices that render and transmit head-tracked binaural audio via wireless technology (e.g., Bluetooth) to binaural audio devices (e.g., over-ear headphones, AirPods Pro, etc.) worn by the user.

Embodiments may also be implemented in VR/AR systems that include a computing device (referred to as a base station) that renders and transmits head-tracked binaural audio via wireless technology to a head-mounted display (HMD) that provides binaural audio output, or to a separate binaural audio device used with a HMD.

Head-tracked binaural audio rendering is a technique that may be used in applications including but not limited to VR/AR applications to create virtual audio sources that appear stable in the environment regardless of the listener's actual orientation/position.

A head-tracked binaural audio rendering method may render and output a binaural audio stream (including left and right audio channels) to a headset so that the listener hears sounds in a spatial audio sense. In other words, the listener hears sounds as if the sounds were coming from real world locations with accurate distance and direction.

For example, the system may play a sound through the headset so that the listener hears the sound coming from virtual sources on their left, their right, straight ahead, behind, or at some angle. Aspects of the left and right audio channels (e.g., level, frequency, delay, reverberation, etc.) may be attenuated to affect the perceived directionality and distance of a sound.

Apple's patent FIG. 1A below illustrates a user with an iPhone and AirPods Pro using Binaural Audio and 2A illustrates a future head mounted device (HMD) using Binaural Audio VR gaming; FIG. 6B illustrates how different Binaural Audio is, in that it tracks the users head so as to move the audio with the user as if it's always in front of them or projecting sound relative to the user's head so noises behind them sound behind them or to the right of them or above them. Great for RPG type gaming in VR mode.  

Apple's patent FIG. 8 below illustrates providing directionality of sound in multiple dimensions.

Apple notes that while embodiments are generally described in which the rendering device renders multiple audio streams and the headset selects one or more audio streams to provide directionality of sound in one dimension (i.e., the horizontal dimension), embodiments may be used to provide directionality of sound in multiple dimensions, for example to provide sounds at azimuth angles, elevation angles, and sounds to indicate translational movements.

For example, the base station may render audio streams at multiple positions in the horizontal dimension and also render audio streams above and/or below the horizontal dimension.

For example, as illustrated in FIG. 8, the base station may render audio streams at positions A and B in the horizontal dimension and also render an audio stream C above the horizontal dimension.

At the headset, the headset selects and plays the audio stream that is closest to the actual position and elevation (or tilt) of the head based on the most recent head tracking data, or alternatively mixes two or more of the streams if the actual position and tilt of the head is somewhere between the audio streams. For example, the headset may select either A, B, or C if the head position is at or near one of those positions, may mix A and B if the head position is between A and B, may mix A and C if the head position is between A and C, may mix B and C if the head position is between B and C, or may mix A, B and C if the head position is somewhere in the middle. This is all done to provide realistic audio that puts a user in the middle of the action which gamers don't experience today when playing first person shooters for instance.

Last night I watch the movie The Outpost on Apple TV. While the action was at times insane trying to put the viewer in the action, the sound in my headphones was basically flat. The actor moved quickly to impress upon the audience that the enemy was coming from every direction. With the future use of Binaural and Spatial audio, we'll hear the gunfire coming from behind, we'll hear the enemy on the roof just beside us. It will suck us right into the environment as if we were actually there.

Apple's next-gen audio is definitely going to be a game changer for future AirPods Pro and Apple's Head Mounted Display Device if it delivers what it promises.

Apple's patent application that was published today by the U.S. Patent Office was filed back in Q1 2020. Some of the work on this dates back to 2017 – though never made public. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Some of Apple's Inventors

Mr. Merimaa: Audio Algorithms, Audio DSP Engineering Manager

Mr. Eubank: Engineering Manager, Audio, Technology Development Group

Mr. Johnson: Audio technology development