Today the U.S. Patent and Trademark Office officially granted Apple a patent that relates to the field of auralization. More particularly, but not by way of limitation, it relates to techniques for auralization of virtual 3D environments in real-time that includes optimizing acoustical ray tracing processes.

Apple notes in their patent background that over the past few decades, Virtual Reality (VR) and Augmented Reality (AR) technologies have emerged to be powerful tools for a wide variety of applications, e.g., in science, design, medicine, gaming and engineering, as well as in more visionary applications such as the creation of "virtual spaces" that aim to simulate the look and sound of their real-world environment counterparts. However, most of the innovation in recent years has been focused on creating virtual visual renderings (e.g., VR headsets and video gaming systems, and the like). In order to increase the sense of immersion in such virtual environments to be as realistic of a simulation as possible, it is important to consider multiple sensory stimuli beyond just the simulation of visual stimuli, e.g., the simulation of sound stimuli--and even smell and/or touch stimuli.

Analogous to visualization, the so-called "auralization" of virtual environments describes the simulation of sound propagation inside enclosures, where methods of Geometrical Acoustics (GA) may be used for a high-quality synthesis of aural stimuli that mimic certain realistic behaviors of sound waves. In such simulations, spatial audio signals may be generated that take into account various models of sound wave reflections, as well as models of sound wave reverberations, in three-dimensional environments. Such spatial audio may be generated, e.g., using Digital Audio Workstation (DAW) software or the like, and may be used for various applications, such as room planning and/or musical or architectural sound simulations. Further details regarding the auralization of virtual environments may be found in the co-inventor's Ph.D. thesis: D. Schroeder, "Physically Based Real-Time Auralization of Interactive Virtual Environments," Ph.D. thesis, RWTH Aachen University, 2011 (hereinafter, "Schroeder").

Current implementations of spatial audio synthesis software can often manage the computational load of simulating moving sound sources around a moving receiver in real-time, however, these simulations are often based on a static reverberation. In a real-world scenario, however, there is significant interaction between sound waves and reflective/obstructive surfaces, e.g., when entering or exiting a room. Moreover, various portals in a room (e.g., doors, windows, roofs) may be dynamically opening and/or closing as a user (or virtual user) navigates a real-world (or virtual) environment listening to synthesized audio signals. Each of these changes in a room's architecture or scene composition can have a significant impact on the way that sound waves in the room should be simulated at any given instant in real-time.

There's now a need for improved techniques for the physically accurate auralization of virtual 3D environments in real-time. This includes environments wherein any (or all) of: the sound sources, the sound receiver, and the geometry/surfaces in the virtual environment may be dynamically changing as the sound sources are being simulated. Such techniques may also be applied in Augmented Reality (AR) scenarios, e.g., wherein additional sound information is added to a listener's real-world environment to accurately simulate the presence of a "virtual" sound source that is not actually present in the listener's real-world environment; mixed reality scenarios; sound visualization applications; room planning; and/or 3D sound mixing applications.

Acoustically effective room volume

Apple's granted patent describes methods that are configured to generate a physically accurate auralization of sound propagation in complex environments, including important wave effects such as sound absorption, sound scattering, and airborne sound insulation between rooms.

Such techniques may support not only a realistic sound field rendering, but also the simulation of spatially-distributed and freely-movable sound sources and receivers in real-time, as well as modifications and manipulations of the environment itself.

Techniques may be utilized to determine a more accurate, e.g., "acoustically-effective" room volume. In particular, for certain 3D room models, it is hard to determine the room volume. This is especially true for convoluted spaces, holes (e.g., open windows) or even "half-open" spaces (e.g., stadiums). Many acoustical equations or algorithms require an accurate estimate of the room's (e.g., the environment's) volume.

The acoustically-effective room volume estimation techniques can handle open and half-open spaces, open windows, convoluted spaces with unreachable corners--and can even detect the amount of volume that should be considered behind very small openings. In this way, the same acoustical equations or algorithms already used in the art can be significantly improved by using the acoustically-effective room volume estimate techniques described herein (as opposed to simple 3D model volume calculations).

Use of the acoustically-effective room volume estimate will make these acoustical equations more accurate and more robust to errors, e.g., due to defective or undefined 3D room/environment models.

In addition, techniques may be utilized to perform optimized acoustical ray tracing. Fast and optimized ray tracing algorithms already exist in the graphics/optics field, but current acoustic adaptions either lack performance or physical accuracy. Thus, by considering acoustical wave propagation and the psychoacoustic characteristics of human listeners, the improved ray tracing processes described herein are adapted to the problem of sound (e.g., rather than light) propagation. The improved ray tracer has increased suitability for acoustic purposes and boosts the performance at the same time.

Apple's patent FIG. 1 below shows an exemplary acoustic room model and the corresponding specification of the room model; FIG. 2 is a flowchart illustrating one method of estimating an acoustically-effective room volume.

Apple's patent FIG. 6 below is an exemplary visualization of room acoustics for a virtual 3D environment. The various ray paths represent the reflection paths from a source (e.g., at location #602 in the room) that have been received at a detector (e.g., at location #604 in the room).

(Click on image to Enlarge)

For more details, review Apple's granted patent 11,197,119