Today the U.S. Patent Office published a patent application of Apple's that relates to methods and apparatus for spatial audio navigation that may, for example, be implemented by mobile multipurpose devices. A spatial audio navigation system provides navigational information in audio form to direct users to target locations.

The system uses directionality of audio played through a binaural audio device to provide navigational cues to the user. A current location, target location, and map information may be input to pathfinding algorithms to determine a real world path between the user's current location and the target location.

The system may then use directional audio played through a headset to guide the user on the path from the current location to the target location. The system may implement one or more of several different spatial audio navigation methods to direct a user when following a path using spatial audio-based cues.

To guide the user on a path while listening to music, the apparent source of the music may be placed in front of the user to guide the user along the path, and moved to the side of the user to prompt the user to make a turn on the path. Thus, the music is not interrupted, and embodiments provide a subtler method to convey navigational information than conventional navigation applications.

In addition, psychologically a user may tend to assume vocal directions are correct and thus follow the directions without much thought, potentially causing accidents. By using directionality and distance of sound as an audio cue to guide a user instead of vocal directions to tell a user, it is up to the user to determine if it is safe to follow the directional audio cue. A voice does not tell the user to do something (e.g., “turn left” or “turn right”); instead, the user is following the directional audio cue. When following something such as a directional audio cue, a different psychology comes into play than when listening to spoken commands.

The spatial audio navigation system may output audio to the headset via a wired or wireless connection so that the user hears the sound in a spatial audio sense. In other words, the user hears the sound as if the sound were coming from a real world location with accurate distance and direction. For example, the system may play a sound through the headset so that the user hears the sound coming from their left, their right, straight ahead, behind, or at some angle. The direction that the sound seems to be coming from may change to guide the user on the path. For example, as the user approaches a left turn on the path, the sound may be moved to the left by increasing the volume of the left audio output channel and/or by decreasing the volume of the right audio output channel. In some embodiments, the system may modulate the volume of the sound to give the user a sense of distance, for example to make it seem as if the sound was coming from three feet, ten feet, or fifty feet away. Instead of or in addition to adjusting the volume, other aspects of the left and right audio channels may be attenuated to affect the virtual directionality and distance of the audio, including but not limited to frequency and reverberation. In some embodiments, the system may change the sound based on detected events, for example by playing an alert or warning sound if the user has missed a turn or is approaching an obstacle.

Embodiments of the spatial audio navigation system may implement one or more of several different spatial audio navigation methods to direct a user when following a path using spatial audio-based cues. The methods may include, but are not limited to, the following:

Continuous path tracing—A sound is moved such that it continuously follows the path. When the sound reaches the end of the path, it returns to the beginning of the path and the process repeats.

Discrete path tracing—A sound is moved in discrete intervals along the path. When the sound reaches the end of the path, it returns to the beginning of the path and the process repeats.

Local direction of currently playing sound—The user is listening to an audio source such as music, a podcast, or a phone call. Instead of the user hearing the audio source as simple stereo sound through the headset, the spatial audio navigation system positions the sound at some distance away from the user in the direction that the user should move to follow the path. The sound may be moved to the left or right to follow bends or turns in the path.

Sound directionality—A sound moves from behind to in front of the user towards the direction that the user needs to move to follow the path, stops, and then repeats. The sound is positioned at some distance away from the user's head, and may be positioned either on their left or right, for example depending on ear dominance.

Sound tunneling—Ambient noise in the environment is sampled or simulated. For example, if the user is in a forest the ambient noise may include the rustling of trees, or if the user is in a city the ambient noise may include crowd and traffic sounds. This ambient noise is then played by the spatial audio navigation system. Obstacles to the sound are placed by the spatial audio navigation system that completely surround the user except in the direction that the user should move to follow the path. This causes a sound tunneling effect that prompts the user to follow the path in the direction of the ambient noise.

Ambient sound occlusion—As above, the appropriate ambient noise in the environment is sampled or simulated by the spatial audio navigation system and played to the user. However, an obstacle is placed in the direction the user should move to follow the path and oriented to face the user. This blocks out the ambient noise in the direction that they should move, causing an effect where there is a gap in the ambient noise that prompts the user to follow the path in the direction of the absence of or gap in ambient noise.

Sound chasing—A sound (e.g., the sound of a pack of wolves, train, etc.) is played at some distance behind the user; as the user moves, the sound “chases” or follows the user to push the user along the path.

Apple's patent FIG. 6C below illustrates a discrete path tracing method in which a sound is moved in discrete intervals along the path. In some embodiments, when the sound reaches the end of the path, it returns to the beginning of the path and the process repeats.

(Click on image to Enlarge)

While embodiments are primarily described in which the spatial audio navigation system is implemented in a mobile multipurpose device connected by a wired or wireless connection to a headset worn by the user, embodiments of a spatial audio navigation system may also be implemented in a mobile multipurpose device or other portable computing device such as a notebook or netbook computer without a headset in which two or more speakers of the device are used to output the navigational information in audio form, and thus serve as the binaural audio device.

In addition, embodiments may be adapted to work in vehicle A/V systems in which vehicle location and orientation technology is used to determine location, motion, and orientation of the vehicle and in which the vehicle's “surround” speaker system is used as the binaural audio device to output the directional sounds to guide the user while driving to a target location, and in homes or other buildings or environments in which a mobile multipurpose device or other technology is used to determine the location, orientation, and movement of the user while a “surround” speaker system is used as the binaural audio device to output the directional sounds to guide the user.

Some of Apple's Inventors

• Stephen E. Pinto: (Position unknown)
• Rahul Nair: Vision Products, Software Engineer
• Ing-Marie Jonsson: UX Researcher at Apple