Apple's late CEO Steve Jobs told Biographer Walter Isaacson that he'd "like to create an integrated television set that is completely easy to use. It will have the simplest user interface you could imagine. I finally cracked it." In that very same timeline, Apple's engineers were working on his dream TV and in a patent application published this morning, we get a tiny glimpse of the enhanced audio system that may be accompanying this future HDTV. The system will also intelligently integrate surround sound-like quality for video conferencing for the home or work. Apple's TV will also be able to work with FaceTime and create a unique audio experience whereby the voices from multiple participants will be heard in the exact direction that they're positioned on the screen. While it's not the Killer HDTV patent we're all still waiting for, it's the next best thing. Today we could safely say that Apple's Killer HDTV is real, at least on paper.
Apple's Patent Background
Electronic devices, such as computers, mobile phones, audio players, laptops, tablet computers, televisions (hereinafter an "electronic device") typically may have an integrated audio output device (e.g., speakers) or may be able to communicate with an audio output device. Additionally, many electronic devices may also include a visual or video output device or communicate with a video display device.
Many audio/visual output devices may be able to have an improved audio or video output, if the audio output is able to be adjusted to the environment, surroundings, circumstances, program, and/or environment. However, many audio and video output devices may require a user input or interaction in order to change a particular output or may not have variable output settings. In these instances the audio and/or video output may not be performing or outputting the best quality sound or images for the particular environment, programs, circumstance, or the like.
Apple's invention may take the form of a method for outputting audio from a computing device. The method may include detecting a user by a sensor. Once a user is detected, a process determines whether the user is an optimum range for a current audio output of an audio output device. If the user is not within the optimum range, the processor modifies the audio output. Additionally, the sensor determines whether the user is orientated towards the computing device. Based on the user orientation the processor adjusts an audio device.
Other examples in Apple's invention may take the form of a method for enhancing audio for a computer. The method may include determining by a sensor a user location relative to the computer. Once the user location has been determined, the sensor may gather environment data corresponding to an environment of the computer. Then, a processor adjusts an audiovisual setting view of the environment data and the user location.
Still other examples in this patent filing may take the form of a system for enhancing audio including a computer and an output device. The computer includes a sensor configured to determine a user location relative to the computer. The sensor is also configured to gather environment data corresponding to an environment of the computer. The computer also includes a processor in communication with the sensor and configured to process the user location and the environment data and adjust at least one of an audio output or a video output. The output device is in communication with the processor and is configured to output at least one of the audio output or the video output.
Apple's Exemplary Systems Includes an HDTV
In an exemplary embodiment, the invention may take the form of a system for providing an enhanced audio experience for a user. The system may include a computer or other electronic device and audio output devices (which may be integrated, separate or a combination of both from the computer 102). The computer may be substantially any type of electronic device with processing capabilities, including, but not limited to, a laptop, tablet, smart phone, audio player, and television.
Apple's patent FIG. 2 illustrated below is a block diagram of the system illustrating an exemplary audio/video processing paths from input to output. In one example, the video sensor 134, sensors 124, and audio input 132 may provide image data regarding the user and/or the environment (e.g., room, surroundings) of the computer or television. The processor 118 may then enhance or alter the audio output characteristics provided to the speaker 110 to provide an enhanced audio experience.
The way the audio output may sound to a user may be dependent on or affected by where a user may be located with respect to the audio output device, as well characteristics of the room or environment. If the audio characteristics or settings are not altered, an audio signal that may have a particular sound in a first room may sound drastically different in a second room.
For example, if the first room is smaller than the second room or if the first room has carpet and the second room has wood flooring.
Advanced Video Conferencing for Work or at Home
In Apple's patent FIG. 5A shown above we see an example of the computer and/or television displaying a multi-person video conference. As can be seen, the display included images of Person A, Person B, Person C, and Person D. Patent FIG. 5B is a top view of the computer/television relative to the positions of each of Persons A-D. As shown in FIG. 5B, each Person A-D is positioned at a different location with respect to the computer/television.
According to Apple, the microphones on the computer/television may also pick up the voices and other sounds of the Persons A-D relative based on their position to the computer/television. Therefore, Persons A and B may be shown smaller compared to Person C on FIG. 5A and the sounds from Persons A and B may also be generally outputted as quieter than the sounds from Person C.
In other embodiments, the system may be configured to alter an audio output based on a location of a video conferencing window or instance. Apple's patent FIG. 7A illustrates the computer/television with an instant messaging, voice, or video chat program running. For example, the computer/television may be running a program such as Apple's FaceTime. The display may include multiple chat instances 430, 432, 434, 436 with each window including its own audio (Audio A, Audio B, Audio C, and Audio D) respectively. Additionally, each chat instance may include a corresponding image, such as a video image or a photograph.
Apple's patent FIG. 7B above illustrates the audio direction for Audios A-D that correspond to chat instances 430, 432, 434, 436 respectively. As can be seen in FIG. 7B, the audio may be directed towards the user in a manner that may correlate to the chat instance location on the display 104. This may allow the user to hear the audio from any of Audios A-D as though they were output from the location of the chat instance. This may provide a more realistic chatting experience of the user in a video conferencing or video chat application. Additionally, the directional audio may also enhance the audio experience of the user during the chat session.
In a video conferencing scenario, the audio processing unit may direct or steer the microphone towards a particular user speaking to better capture their voice. Similarly, the image processing unit may focus or zoom the video sensor on a particular user. In still other examples, the user/interface processing unit may direct particular sensors to gather additional environmental/user data. Additionally, the output processing may include frequency filters to post-process an audio signal (e.g., to reduce noise frequencies, enhance particular frequencies, and so on), correct errors in audio levels, adjust loudness to a particular level (e.g., equalize an audio output), echo-cancellation, peaking filters and so on.
The audio may be adjusted for the user so that the volume may be increased, the external speakers may be turned on, internal speakers turned off, surround sound may be switched from a "screen channels" setting into a surround sound format, or the surround sound channels may be redirected from internal speakers to external speakers and to left-surround channels and right-surround channels. On the other hand, if the audio is already adjusted or configured with a distance setting, the audio may not need to be adjusted based on the user's location.
Apple states that the external speakers may form a part of a surround sound speaker array and therefore may provide a better "room" sound experience as compared with the internal computer or television speakers.
Facial Recognition and Eye Tracking
Apple states that the image processing unit may utilize gaze tracking to track the gaze of the user's eyes, facial recognition or other similar methods to determine if the user's head direction with respect to the computer or television. Further, if the user has been utilizing the microphone (e.g., for a phone call, video chat, dictation), the computer or television may steer the microphone towards the direction of the user's mouth. Now that's pretty cool.
Apple credits Aleksandar Prance, Brett Bilbrey, Darby Hadley, Martin Johnston and Ronald Isaac as the inventors of this patent application which was originally filed under serial number 193461 in Q3 2011. It should be noted that back in 2006 an Apple invention came to light about tiny cameras being integrated into the very fabric of a future display. This would allow for true eye-to-eye video conferencing. Today's concept working with enhanced audio would provide the ultimate videoconferencing system on the planet and one that could very well be integrated into a future HDTV. Now that would be a killer application that would rock the TV world.
Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.
New on Patent Bolt this Week