The Problem to Solve

With their small size and portability, portable media devices could be used during a variety of activities, for example while reading, walking, running, biking and driving. While engaged in these activities, the user could be "visually" focused on the activity and simultaneously "aurally" connected with the portable media device. Interacting with the device without using a visual interface could be accomplished with limited functionality on today's devices, such as by depressing a simple on/off button and operating a volume control on a headset connected by a wire or wirelessly to the device.

More complex interactions with the device, such as searching for and selecting a particular stored media file could require the user to disengage visually from the current activity to use a displayed graphical user interface on the device. The graphical user interface could include control inputs, e.g. a touch screen, touch pad or a scroll wheel button by which the user could navigate through the stored media libraries on the device. Using the graphical user interface, however, could disrupt the activity in which the user is otherwise visually engaged, and some portable media devices do not include a visual display. Thus there exists a need for an alternative user interface that enables the user to interact with and control the portable media device to navigate through and retrieve from a set of media stored in the device.

Apple's Solution

Apple's patent relates to systems, methods and apparatus that enable a user to interact with a portable media device through a unique audio user interface. In described embodiments, the audio user interface could include one or more audible control nodes perceived to be spatially located at different points about the user of the portable media device. A sensor in the portable media device could sense a movement of the portable media device by the user toward one or more of the audible control nodes. The operation of the portable device could be modified in accordance with the sensed movement of the portable media device.

Operating a Media Device via Three Dimensional Space

Apple's patent FIG. 1A illustrates a user positioned in a three dimensional reference coordinate system 10 listening to portable media device 12 through headset 14. Apple's patent FIG. 1B illustrates a portable media device that could include audio output circuit, data storage device, processor, and more importantly, a sensor and user input interface.

The user could move their portable media device in response to hearing the audio user interface in order to alter an operation of their portable media device. For example, moving portable the media device in the direction of one of the audio control nodes 16 (or executing some other translational or rotational movement) could result in the portable media device changing volume of a media item being played. More broadly, the portable media device could sense a movement and associate the sensed movement with a specific operational change that could modify a specific operation of the device. The sensor 108 could include, without limitation, an accelerometer, a gyroscope or another motion sensing device that could detect a change in position, orientation or movement of a portable media device along a number of different reference directions as shown above in patent FIG. 4.

Menus Presented Based on Spatial Direction or Text-to-Speech

Apple's patent FIGS. 5A and 5B illustrate two sets of menu items of an audio user interface presented at spatially distinct locations with respect to a user's position.

These menu items could be presented in a variety of forms such as synthesized text-to-speech, recorded speech or samples of recorded music. In some embodiments, the menu items could be presented sequentially, such as when articulating the menu items as speech, or could be presented simultaneously, such as when communicating the menu items as samples of recorded music. (Speech could also be presented simultaneously and recorded music samples sequentially as well.)

The user could move portable their media device in the direction of a menu item, and the sensor could detect the movement. The processor in the device could determine, based on the sensed movement, the menu item indicated by the user and present a confirmation of the indicated menu item to the user, such as repeating the menu item or sample of recorded music.

The user could then select the indicated menu item by a second movement of their portable media device, for example a quick shake, or by confirming through a user input interface, such as depressing a physical button, on your portable device. In some embodiments, the portable media device could automatically select the indicated menu item based on the initial sensed movement without using a confirmation from a second gesture.

Mapping Multiple Device Movements to Specific Actions

Apple's patent FIG. 6 illustrates different movement gestures that could be associated with selecting a menu item.

The mapping of movements/gestures to an intended selection could be "many to one," i.e. several different movements could be used to indicate the same selection. A number of different movements could be associated with an indicated action as shown in FIG. 6. A "centered" position (610) of a portable media device could indicate playing a "current" selection (604), such as a "current" track within a particular album.

The user could displace the portable media device along a left/right axis (614) to choose a "previous" selection (602) or a "next" selection (606). In this case, a "left" position (608) along the axis (614) can indicate "previous" (601) and a "right" position (612) along the axis could indicate "next." Thus a sensor within the portable media device could sense position, and the processor could change a selection according to sensed position.

Media Menus Controlled by an Invisible Spatial Arc

Now this is wild. Apple's patent FIG. 8A illustrates an audio user interface presented by a portable media device through a stereo (binaural) headset (804) where the set of menu items are arrayed spatially in an arc about a user (top view).

Three selections of jazz genre music could be presented either sequentially or simultaneously to the user. A segment of Miles Davis music could be presented centrally and "in front" of the user, while a segment of John Coltrane music could be presented to the left of the user and a segment of Dave Brubeck music presented to the right of the user.

Audio processing algorithms could be applied to the "left" music and "right" music so that they appear more distant (e.g. quieter) and "off center" compared to the "center" music. Changing the position of the device or specific movements of the device could also alter the combined sound from all three locations presented to the user. For example moving the device from left to center to right could cause each of the directionally indicated music to play louder while the others played softer.

Alternatively, one could consider the arrangement shown in FIG. 8A as an aural form of a visual user interface known as "cover flow" used in Apple products such as the iPod. While FIG. 8A illustrates only three menu selections at left, center and right positions, additional menu selections could also exist off stage. In one embodiment the menu selections could continue in the right and left directions continuously, with the user able to shift the center position to a present a new menu selection by a particular movement of the portable media device.

Audio Driven Spatial Rooms for the Multitasking Music Freak

In Apple's patent FIG. 8B noted above we see another embodiment of an audio user interface presented by the portable media device to a user through the headset (804 top view). A set of menu selections could be spatially processed in the portable media device to resemble sitting within a series of connected rooms 812, 814, 816.

A segment of jazz music could be presented as the current selection with the user seeming to be situated within a "jazz" room 812, while simultaneously the user could hear a segment of rock music emanating from an adjacent room 814 to the left and a segment of country music emanating from an adjacent room 816 to the right. A processor in the portable media device 802 could spatially process the music to diminish the volume and "occlude" the sounds from the adjacent "rooms" to represent distance from the user and intervening room walls. Moving the device to the left or right could cause the sound field to shift appropriately (for example the user could seem to switch position to adjacent rooms).



Today's published patent application 20110153044 from Apple was originally filed in Q4 2009 by inventors Aram Lindahl and Richard Powell.

Notice: Patently Apple presents only a brief summary of patents with associated graphic(s) for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for further details. Patent applications shouldn't be digested as rumors or fast-tracked according to rumor time tables. Apple's patent applications represent true research that could lead to future products and should be understood in that light. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.