On the last patent application day of the year, the USPTO revealed a real treat for us. It appears that Apple has added yet another dynamic new chapter to their ongoing smart bezel project; a project that has the potential of taking Apple's iDevices to the next level. It could, without a doubt, revolutionize Apple's entire tablet-based line-up in the blink of an eye. The idea revealed in today's patent application is to advance iOS devices with a new live interface that is referred to as the Multi-Modal Human Interface. The new user interface is powered by a new engine that is able to detect environmental conditions and change the operational interactivity options for users so as to maximize the iDevice's usefulness - automatically. It's also designed to reduce power drainage so that devices could be up and running longer. One example provided for in Apple's documentation is an iPad shutting down the standard iOS UI automatically in favor of one that is driven by a voice and speech recognition UI while the user is driving. This is certainly another natural avenue for Apple's Siri. Yet, in the big picture, the smart bezel project is all about advancing the usability of future iOS devices on multiple levels. It's a project that demonstrates that Apple's leadership in all-things-tablets is based on driving the boundaries of true innovation to market and in the faces of their never ending competition. And on that note, I say Cheers to the Crazy Ones of Cupertino!
Apple's Patent Background
Apple's invention background points to the iPad, iPhone and iPod touch having to have a lightweight battery yet have enough power to use the device and store information. In order to maximize an amount of time that an iOS device could operate while powered by the battery, the power consumption of the iOS device must be optimized for current operating conditions.
Depending upon the display technology, the display could consume a substantial amount of power when active. This is especially true of transmissive type displays (such as liquid crystal display, or LCD) that require an illumination source (also referred to as a backlight) that have a particularly high power demand. However, LCDs could be manufactured to be lightweight and thin, making them eminently well suited for use in small form factor portable electronic devices in spite of their high power requirements.
Therefore, a system, method, and apparatus for maintaining the ability of a user to interact with an electronic device for longer times are desired.
Apple's Solution
Apple states that their invention covers small form factor electronic devices that include a processor and an interface engine in communication with the processor and a sensor coupled to the processor. The sensor is arranged to detect at least one environmental factor and pass an indication of the detected environmental factor to the processor.
The processor and the interface engine cooperate to determine if an environment of the electronic device has changed, identify an updated human interface when the environment has changed, and cause the small form factor electronic device to present the updated human interface only if a level of interactivity corresponding to the updated human interface is at least greater than a threshold level of interactivity.
For example, if the electronic device is battery powered and presents visual information using a transmissive type display (that requires backlight illumination), then providing an alternative human interface that relies upon enhancing the presentation of the visual content by increasing the amount of light (and therefore power) provided by the backlight in an environment of high ambient light (such as sunlight) is counterproductive to the maintenance of a long battery life.
Shifting to Non-Visual Modes to Preserve Battery Life
Accordingly, the alternative human interface in the bright sunlight scenario should rely on mechanisms other than increasing the backlight in part or in whole to maintain the minimum level of interactivity as well as the useful life of the battery. For example, the alternative human interface in a bright light environment could rely upon non-visual interface modes such as a haptic interface mode, an audio interface mode, an inertial interface mode, and so on used singly or in combination.
Apple's introduces us to the Multi-Modal Human Interface (MMHI)
Apple states that it should be noted that the environmental factors considered when determining an appropriate alternative human interface could include those external to the electronic device such as ambient light, ambient sound, and context of use. The environmental factor considered could also include those internal to the electronic device such as battery level and display technology. The embodiments described are particularly well suited for small form factor battery powered electronic devices having a display and could be implemented automatically based upon the detection of specific environmental factors at predefined levels.
In a particular embodiment, the alternative human interface could take the form of a multi-modal human interface (MMHI) provided by an MMHI engine. The MMHI engine could provide an updated MMHI arranged to automatically maintain a pre-determined level of interactivity between a user and the electronic device. In addition to maintaining at least the pre-determined level of interactivity, the updated MMHI could preserve selected operational resources, such as battery charge, when the electronic device takes the form of a portable computing device powered by a battery.
Apple's patent FIG. 1 shown below illustrates a simplified block description of multi-modal human interface (MMHI) engine.
An iPad with a Multi-Modal Human Interface Engine
Apple's patent FIG. 2 shown below illustrates an iPad device having an associated MMHI engine.
As the situation may require, being able to update the MMHI (108) in such a way that a user could interact with an iPad (or other iOS device) without having to rely on visual indicators could be very useful. For example, it would be advantageous to not rely on visual indicators provided by display when the display isn't viewable or that using the display would adversely affect the operation of the iPad by, for example, severely reducing expected operating time at a current operating state.
The reduction in expected operating time could be caused by many factors such as the inordinate power drain required to support a transmissive type display in an environment of high ambient light. For example, as well known in the display arts, transmissive displays rely upon an external light source (referred to as a backlight when used in the context of a liquid crystal display, or LCD) to provide images for viewing. The backlight could require substantial amounts of power to operate in even the most optimal light conditions. However, when light conditions change (going from dark to bright, for example) there may not be sufficient power resources available to drive the display in the bright environment sufficient to overcome the ambient light conditions. The bright light could "wash" out any images presented on the display severely restricting the ability of a user to interact with their iPad.
In order to maintain the ability of the user to interact with their iPad as well as maintaining battery charge and therefore expected battery life, the MMHI could be updated in such a way as to not rely on the display (or at least substantially reduce the reliance on the display.
Interactivity previously provided by the display (and more precisely the display and touch sensitive layer 204) could be provided in the alternative by other interface mechanisms acting singly or in cooperation with each other. For example, with the display effectively out of the loop so to speak, haptic actuator 124 and/or speaker 126 could be used separately or in tandem to provide a human interface that at least preserves the ability of a user to interact with their iPad as well as preserve battery charge. The preservation of the battery charge could also prolong useful operation of the iPad than would otherwise be possible using the display as the primary mode of interaction.
While the invention's documentation is focused on some exacting examples, we're able to recognize that what Apple is actually trying to achieve here, is very much in sync with their Smart Bezel project. In fact our April 2011 report is a great point to refer back to in order to get the bigger picture. In fact, the current patent by itself is very cryptic and very much an island to itself until you realize it's a part of a much larger context of the Smart Bezel project. See our other reports to get the bigger picture: One, two, three, and four. The latter covers Apple's consideration of implementing a hybrid type of display to in order to conserve display energy for future iDevices.
Apple only hints of the smart bezel concept ever so indirectly in this current patent by pointing to the bezel's multi-modal interface being located in the bezel area, which is also supported by patent point 122, the display. The display is noted as being both the main display area of a typical iPad as well as being represented in the bezel. For technical and perhaps legal reasons, Apple has to separate their various smart bezel technologies into distinct segments that are deadly focused. But for us the consumers trying to understand it, you need to remember the context of the Smart Bezel Project.
Automatically Switching UI Modes Conditions
Apple's patent FIGS. 3-5 noted below graphically illustrate various configurations of the MMHI based upon representative local environments detected by the sensor and evaluated by the processor in cooperation with the MMHI engine. For example, patent FIG. 3 shows us a situation whereby the iPad is exposed to an environment having little or no ambient light, such as at night. In Patent FIG. 4 we see a situation where the local environment of the iPad has changed from one of low ambient light of FIG. 3 to one of high ambient light as one would expect from bright sunshine. In each instance where the iPad's environmental conditions change, we see that the iPad's future MMHI will provide optional interfaces to assist the user better interact with their iPad.
In Apple's patent FIG. 5 noted above we see an illustration where the MMHI is configured to operate in "driving mode" when the sensor detects that the iPad is moving at a greater than threshold speed, such as for example, 20 mph indicating motion in a moving vehicle. In this situation, the processor and the MMHI could cause the MMHI to rely upon a more limited range of interface modes consistent with safe driving (the presumption being that the user is driving the vehicle, be it car or bike) and the iPad should not provide a distraction. In this way, the MMHI could be configured by the processor in cooperation with the MMHI engine to rely upon the audio interface mode (think Siri) by way of a speaker and/or the speech/sound recognition mode and microphone in order to maintain user interactivity in a safe driving environment.
Apple's patent application was originally filed in Q2 2010 by inventors Jesse Boettcher, Jeffery Lee and Aram Lindahl.
An Interesting Patent Application Twist Published Today
For Interest Sake: One of Apple's continuation patents that were published today dates all the way back to 2007. Interestingly, this patent officially lists Apple as: Apple Inc., a Delaware corporation. Was it in error or is there an alternative explanation? If you think that you have a plausible explanation for this, send in your comments.
Notice: Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. Apple's patent applications have provided the Mac community with a clear heads-up on some of Apple's greatest product trends including the iPod, iPhone, iPad, iOS cameras, LED displays, iCloud services for iTunes and more. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.
Here are a Few Sites covering this Original Report: MacSurfer, Twitter, Facebook, Apple Investor News, Google Reader, Macnews, iPhone World Canada, MarketWatch, The Unified View, Unified Communications Strategies, and more
