Patently Apple has covered a number of spatial audio patents from Apple since 2020 (01, 02, 03, 04 and more) covering it's use in AirPods Pro, in a future VR Headset, supporting VR gaming and more. Today the US Patent & Trademark Office published a continuation patent from Apple titled "Spatial Management of Audio."

Apple's patent covers techniques for managing spatial audio using electronic devices, however, are generally cumbersome and inefficient. For example, some techniques do not provide users with contextual awareness of the state of the electronic device through spatial management of audio.

For another example, some existing techniques use a complex and time-consuming user interface, which may include multiple key presses or keystrokes. Existing techniques require more time than necessary, wasting user time and device energy. This latter consideration is particularly important in battery-operated devices. Further, existing audio techniques to not sufficiently help users navigate graphical user interfaces.

Accordingly, the present technique provides electronic devices with faster, more efficient methods and interfaces for managing spatial audio. Such methods and interfaces optionally complement or replace other methods for managing spatial audio.

Such methods and interfaces reduce the cognitive burden on a user and produce a more efficient human-machine interface. For battery-operated computing devices, such methods and interfaces conserve power and increase the time between battery charges.

At FIG. 13A below, an iPhone (#1300) displays a music player (#1304), which includes album art (#1304a) and, optionally, axes (#1304b). The album art corresponds to a song being played by an iPhone. In some examples, the song includes a plurality of audio streams. In this example, the song includes five audio steams, where each audio stream corresponds to a particular instrument.

At FIG. 13A, an iPhone produces, using the left (#1358a) and right (#1358b) speakers, audio for the song (including all five audio streams) without placing the audio in space (e.g., an iPhone doesn't apply any of an interaural time difference, an HRTF, or cross cancellation).

For example, this results in the user perceiving the audio as being in the head of user, as illustrated by audio element (#1354) in overhead view 850 of FIG. 8F-8G. the audio element corresponds to the song, including the five audio streams. An iPhone is configured such that user inputs received at affordances (#1304c) control the music, such as by pausing, playing, fast forwarding, and rewinding the music.

In patent FIG. 13H below we're able to see how spatial audio can separate each instrument in 5 distinct streams to cover a singer, a keyboard, drums, bass guitar and rhythm guitar.

Apple's spatial audio patent was first granted to Apple in August 2019. Today's patent is a "continuation patent" which is like a patent update wherein Apple is able to update technology and/or secure specific features by providing additional patent claims. In today's continuation patent Apple has added 13 new patent claims supporting spatial audio. Below are just a few of the new patent claims:

New Patent Claim #1: "An electronic device, comprising: a display; one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, wherein the electronic device is operably connected with two or more speakers, the one or more programs including instructions for: displaying, at a first location on the display, a first visual element; accessing first audio corresponding to the first visual element; while displaying, at the first location on the display, the first visual element: producing audio, at the two or more speakers using the first audio, in a first mode; and receiving first user input; in response to receiving the first user input: transitioning display of the first visual element from the first location on the display to the first visual element not being displayed on the display; while not displaying, on the display, the first visual element, producing audio, at the two or more speakers, using the first audio in a second mode that is different from the first mode, wherein the second mode is configured such that audio produced in the second mode is perceived by the user as being produced from a direction that is off the display; and transitioning from not producing audio using second audio that corresponds to a second visual element to producing, at the two or more speakers, audio using the second audio that corresponds to the second visual element."

New Patent Claim #2: "The electronic device of claim 1, wherein the first mode is configured such that audio produced using the first mode is perceived by the user as being produced from a first direction that corresponds to the display."

New Patent Claim #3: "The electronic device of claim 1, wherein the one or more programs further include instructions for: accessing the second audio corresponding to the second visual element; while displaying, at the first location on the display, the first visual element and while not displaying the second visual element: producing audio, at the two or more speakers using the second audio, in a third mode different from the first mode and the second mode, wherein the third mode is configured such that audio produced in the third mode is perceived by the user as being produced from a direction that is off the display; and forgoing displaying, on the display, the second visual element that corresponds to the second audio."

New Patent Claim #4: "The electronic device of claim 1, wherein the one or more programs further include instructions for: while displaying, at the first location on the display, the first visual element: forgoing displaying, on the display, the second visual element that corresponds to the second audio; and in response to receiving the first user input: transitioning display of the second visual element from not being displayed on the display to a fourth location on the display; and concurrent with producing audio using the first audio in the second mode, producing audio, at the two or more speakers, using the second audio in the first mode."

To review the remaining 9 new patent claims and the patent in general, review continuation patent 20210232359.

Other Key Continuation Patents

It's a slow summer for patents and this is a time when continuation patents stand out most. Below is a list of other key continuation patents published today by the U.S. Patent Office.

Focusing for Virtual and Augmented Reality Systems

Apple's continuation patent covers methods and apparatus for focusing in virtual reality (VR) or augmented reality (AR) devices based on gaze tracking information are described. Embodiments of a VR/AR head-mounted display (HMD) may include a gaze tracking system for detecting position and movement of the user's eyes.

For AR applications, gaze tracking information may be used to direct external cameras to focus in the direction of the user's gaze so that the cameras focus on objects at which the user is looking.

Apple's patent FIG. 2 below shows a side view of an example HMD that implements a gaze tracking system; FIG. 1E illustrates focusing eye lenses in VR HMDS based at least in part on gaze tracking information.

Apple was granted this patent on April 19, 2021. Today, the U.S. Patent Office published Apple's continuation patent and revealed that Apple scrapped their original 20 claims and replaced them with 20 new claims continuously numbered 21 - 40. Below are just a few of the key new claims:

Patent Claim #21: "A system, comprising: a head-mounted display (HMD) that displays visual content for viewing by a user, wherein the HMD comprises: one or more external cameras that capture video of an environment; at least one display panel that displays frames of visual content based on the video of the environment captured by the one or more external cameras for viewing by the user; and a gaze tracking system comprising left and right eye tracking cameras that capture infrared or near-infrared light reflected by the user's eyes; and a controller comprising one or more processors, wherein the controller: obtains gaze tracking information from the gaze tracking system; determines direction of the user's gaze from the gaze tracking information; determines distance of an object in the environment in the determined direction of the user's gaze; and records depth information or focus information corresponding to the determined distance of the object, with the frames of visual content, to generate a recorded video."

Patent Claim #25: "The system of claim 24, wherein the one or more external cameras: focus on the object in the environment in the determined direction of the user's gaze; and provide feedback to the controller including the focus information for the object; wherein, to determine the distance of the object in the determined direction of the user's gaze, the controller determines the distance according to the focus information for the object."

Patent Claim #26: The system of claim 21, wherein the HMD plays back the recorded video for viewing by the user, and wherein the controller: directs the left and right optical lenses to adjust focus of the left and right optical lenses according to the depth information or the focus information for the object so that optical vergence of the displayed object matches convergence of the user's eyes.

Patent Claim #27: "The system of claim 26, wherein the left and right optical lenses form a virtual image of the frames displayed on the at least one display panel at a distance at or near optical infinity of the optical lenses, and wherein, to direct the left and right optical lenses to adjust focus according to the depth information or the focus information for the object, the controller directs the optical lenses to reduce refractive power so that the object appears to the user to be at the determined distance instead of at optical infinity."

Patent Claim #29: "The system of claim 28, wherein the HMD plays back the recorded video for viewing by the user, wherein the recorded video includes the virtual content including the second displayed object, and wherein the controller: obtains, from the recorded video, the depth information or the focus information for the virtual object in the virtual reality view in the determined direction of the user's gaze; and directs the left and right optical lenses to adjust focus of the left and right optical lenses according to the obtained depth information or focus information for the virtual object so that optical vergence of the second displayed object matches convergence of the user's eyes."

To review the remaining 15 patent claims and the patent in general, review continuation patent 20210235054.

Other continuation patents published today include: "Optical Systems For Electronic Devices With Displays," (20210231960); iPhone with wireless charging coil (20210232180); Predictive Routing based on Microlocation (20210235361); Virtual Object Kit (20210233329) and "Moving about a Setting (20210232288 filed in 2018) where Apple cancelled 37 old patent claims. In each case, the update is deadly focused on adding new patent claims and nothing else.