(Click on image to Enlarge)

Today the U.S. Patent and Trademark Office officially granted Apple a patent that relates to the field of auralization. More particularly, it relates to techniques for auralization of virtual 3D environments in real-time. Most innovations in AR/VR environments have been centered on virtual 3D environments for gaming. Apple's granted patent covers "auralization" of virtual environments which describes the simulation of sound propagation inside enclosures, where methods of Geometrical Acoustics (GA) may be used for a high-quality synthesis of aural stimuli that mimic certain realistic behaviors of sound waves. Apple is working to advance spatial audio to new dimensions.

Apple notes in their patent background that over the past few decades, Virtual Reality (VR) and Augmented Reality (AR) technologies have emerged to be powerful tools for a wide variety of applications, e.g., in science, design, medicine, gaming and engineering, as well as in more visionary applications such as the creation of "virtual spaces" that aim to simulate the look and sound of their real-world environment counterparts.

However, most of the innovation in recent years has been focused on creating virtual visual renderings (e.g., VR headsets and video gaming systems, and the like). In order to increase the sense of immersion in such virtual environments to be as realistic of a simulation as possible, it is important to consider multiple sensory stimuli beyond just the simulation of visual stimuli, e.g., the simulation of sound stimuli--and even smell and/or touch stimuli.

Analogous to visualization, the so-called "auralization" of virtual environments describes the simulation of sound propagation inside enclosures, where methods of Geometrical Acoustics (GA) may be used for a high-quality synthesis of aural stimuli that mimic certain realistic behaviors of sound waves.

In such simulations, spatial audio signals may be generated that take into account various models of sound wave reflections, as well as models of sound wave reverberations, in three-dimensional environments. Such spatial audio may be generated, e.g., using Digital Audio Workstation (DAW) software or the like, and may be used for various applications, such as room planning and/or musical or architectural sound simulations.

Further details regarding the auralization of virtual environments may be found in the co-inventor's Ph.D. thesis: D. Schroeder, "Physically Based Real-Time Auralization of Interactive Virtual Environments," Ph.D. thesis, RWTH Aachen University, 2011 (hereinafter, "Schroeder").

Current implementations of spatial audio synthesis software can often manage the computational load of simulating moving sound sources around a moving receiver in real-time, however, these simulations are often based on a static reverberation. In a real-world scenario, however, there is significant interaction between sound waves and reflective/obstructive surfaces, e.g., when entering or exiting a room. Moreover, various portals in a room (e.g., doors, windows, roofs) may be dynamically opening and/or closing as a user (or virtual user) navigates a real-world (or virtual) environment listening to synthesized audio signals. Each of these changes in a room's architecture or scene composition can have a significant impact on the way that sound waves in the room should be simulated at any given instant in real-time.

In this way, there is a need for improved techniques for the physically accurate auralization of virtual 3D environments in real-time. This includes environments wherein any (or all) of: the sound sources, the sound receiver, and the geometry/surfaces in the virtual environment may be dynamically changing as the sound sources are being simulated. Such techniques may also be applied in Augmented Reality (AR) scenarios, e.g., wherein additional sound information is added to a listener's real-world environment to accurately simulate the presence of a "virtual" sound source that is not actually present in the listener's real-world environment; mixed reality scenarios; sound visualization applications; room planning; and/or 3D sound mixing applications.

Apple's granted patent covers techniques that may be utilized to determine a more accurate, e.g., "acoustically-effective" room volume. In particular, for certain 3D room models, it is hard to determine the room volume. This is especially true for convoluted spaces, holes (e.g., open windows) or even "half-open" spaces (e.g., stadiums).

Many acoustical equations or algorithms require an accurate estimate of the room's (e.g., the environment's) volume. The acoustically-effective room volume estimation techniques disclosed herein can handle open and half-open spaces, open windows, convoluted spaces with unreachable corners--and can even detect the amount of volume that should be considered behind very small openings. Thus, the same acoustical equations or algorithms already used in the art can be significantly improved by using the acoustically-effective room volume estimate techniques described herein (as opposed to simple 3D model volume calculations). Use of the acoustically-effective room volume estimate will make these acoustical equations more accurate and more robust to errors, e.g., due to defective or undefined 3D room/environment models.

According to other embodiments, techniques may be utilized to perform optimized acoustical ray tracing. Fast and optimized ray tracing algorithms already exist in the graphics/optics field, but current acoustic adaptions either lack performance or physical accuracy. Thus, by considering acoustical wave propagation and the psychoacoustic characteristics of human listeners, the improved ray tracing processes described are adapted to the problem of sound (e.g., rather than light) propagation. The improved ray tracer has increased suitability for acoustic purposes and boosts the performance at the same time.

According to still other embodiments, techniques may be utilized to translate simulated ray tracing results into natural-sounding reverberations that more-accurately account for the laws of physics.

Many acoustic simulation algorithms in the prior art simply perform a direct transformation of the path of a bounced ray into a modeled room reflection. In such simulations, however, a ray cannot represent a reflection. Instead, the number of rays is set arbitrarily at the beginning of the simulation and held constant over time, while the number of room reflections is defined by the room's geometry (and not by the number of rays) and increases exponentially over time.

By contrast, the improved techniques described define a transformation that is used to derive a spatial-time-frequency energy probability density function (PDF) during ray tracing that more accurately accounts for the laws of physics, and then convert this data into a spatial impulse response (SIR) function, which may then be used for realistic 3D audio reproduction, e.g., either via headphones or loudspeakers.

Apple's patent FIG. 1 below illustrates an exemplary acoustic room model and the corresponding specification of the room model; FIG. 6 is an exemplary visualization of room acoustics for a virtual 3D environment.

Apple's patent FIG. 9 below presents a flowchart illustrating another method of performing an improved Spatial Impulse Response (SIR) generation algorithm.

For more details, review Apple's granted patent 11,170,139. One of the listed inventors is Sönke Pelzer, Sr. Engineer - Audio Technology.

You could also review Apple's second granted patent 11,172,320 on this subject matter which share the same patent figures titled "Spatial Impulse Response Synthesis."