The Problem with Traditional Desktop Knob Controls

 

Modern computer operating systems often provide a desktop graphical user interface ("GUI") for displaying various graphical objects. Some examples of graphical objects include windows, taskbars, docks, menus and various icons for representing documents, folders and applications. A user could interact with the desktop using a mouse, trackball, track pad or other known pointing device. If the GUI is touch sensitive, then a stylus or one or more fingers could be used to interact with the desktop.

 

Some computer applications include knobs, sliders and other controls that could be manipulated by a user through a pointing device or by touch input. For example, graphical objects emulating knobs and sliders of an audio mixing board or other hardware device could be included in the GUI of an audio or video application. The user could use a mouse or finger to rotate the knob control or move the slider control as desired to control a parameter of the application.

 

Many GUIs use knob controls to dial inputs up or down. However, traditional GUI knob controls typically have several drawbacks. For example, the interaction model of the knob control could be unclear, such that the user may not know whether to move a mouse pointer vertically, horizontally or in a circle to change the value of the knob control. Additionally, the resolution of a knob control may be less than other types of GUI controls (e.g., a slider control).

 

Apple Introduces a New Hybrid Knob/Slider Control for the Desktop

 

Apple's invention is focused on a new hybrid knob/slider control that could replace conventional knob controls. A control object including a knob element could be displayed in a GUI presented on a display of a device. A first user input could be detected that is indicative of a selection of the knob element. In response to the first user input, the knob element could be visually augmented.

 

The visual augmentation could include displaying an outline and/or fill region (e.g. a color or pattern fill) around the knob element, displaying text labels for minimum and maximum range limit values and displaying a slider element with the knob element. The slider element could include a fill bar to indicate the current value of the knob control.

A second user input could be detected that is indicative of a change of the value of the knob element. The second input could be a linear motion on or near the slider element. In response to the second user input, both the fill bar of the slider element and a fill portion of the knob element could be visually augmented to indicate the change in the value.

The visual augmentation could provide a visual cue of the interaction model for the knob element. For example, the visual augmentation could include displaying a horizontal slider element with the knob element to visually indicate to the user that linear motion along the horizontal or longer dimension of the slider element with a pointing device will manipulate the knob element. The visual augmentation could also provide increased resolution for the knob element by removing the cursor image from the screen and providing an active input region for detecting the linear motion of the pointing device that is wider than the slider element.

 

In some implementations, the slider element could be vertical or arched to indicate linear motion in the vertical direction or circular motion, respectively, to manipulate the knob element. In some implementations, the value of the knob element could also be changed through manual text entry.  

 

Exemplary Hybrid Knob/Slider Controls

 

Apple's patent FIG. 1A shown below illustrates the appearance of an exemplary hybrid knob/slider control in a default state. In the default state, a control object (100) with a knob element (102) is displayed. A parameter name label could be displayed above the knob element (e.g., "Freq 1"), and a current value (e.g., "44") and units label (e.g., "Hz") could displayed below the knob element.

 

The knob element could also include a fill portion 104 (e.g., a "pie slice") showing the current value of the knob element as a percentage of a range, and a line (106) indicating the current value of the knob element. Minimum and maximum range limits for the value are not displayed but could be if desired.

 

 

Apple's patent FIG. 1B illustrates the appearance of the hybrid knob/slider control of FIG. 1A in a first active state. The outline of the knob element could be made brighter to indicate the selected state and text labels for minimum and maximum range limits (e.g., "0.5" and "8000") can be displayed.

 

Apple's patent FIG. 1C illustrates the appearance of the hybrid knob/slider control of FIG. 1B in a second active state. In the second active state, a user has mouse-down or touch clicked on the control object. By doing so, the control object is augmented with an outline (108) and a fill region (109) at least partially surrounding the knob element. The fill region could be a color, pattern, shadow or a combination of these visual properties.

 

The Slider Element

 

A slider element (110) is displayed beneath the knob element. In some implementations, the slider element could be displayed along a side of the knob element. The slider element could have a first dimension and a second dimension, where the second dimension is longer than the first dimension (e.g., a square rectangle or rounded rectangle).

 

The slider element could also provide a visual clue of the interaction model for the knob control, which in this example is linear motion in the horizontal or +/-x direction. Because, in this implementation, the slider element is only present when the control object is selected or active, screen space is conserved, which could be advantageous for portable devices with smaller screens or applications that require a dense array of control objects or groups of control objects (e.g., a virtual mixer board).

 

Audio Application Example

 

 

Apple's patent FIG. 2A illustrates a group of knob elements (206) in a first active state. The group of knob elements includes four knob elements, which could be used in an audio application. In the first active state, the user has moused-over the group of knob elements and the mouse pointer 214 is visible.

 

Apple also notes that the "Balance" parameter knob element noted in FIG. 2B is shown with a current value of 50%. If this knob element were active and manipulated by the user, then a bipolar fill bar 212 could be used to indicate the current value of the knob element for the "Balance" parameter.

 

Virtual Audio Compressor Example

 

 

Apple's patent FIG. 3A illustrates a group of knob elements in a default state. The group includes a three knob elements, which are typical for a virtual audio compressor. In this example, a knob element 302 could include text labels indicating a parameter name and current value/units of the knob element. In addition to the text labels, a fill portion (304) shows the current value as a percentage of the allowable range of the knob element and line 306 provides the current value. Patent FIG. 3B illustrates "the active state," as described earlier in patent FIG. 1C.

 

In some implementations, the slider element 302 could be displayed vertically along the left or right side of the knob element. In such an implementation, a user's vertical movement of a mouse pointer, stylus or finger along the slider element changes the value of the knob element. Let's be clear here, Apple isn't talking about large trackpad here, they're talking about a touch display or touchscreen in context with a desktop.

 

The group of knob elements could apply to an audio application, where the knobs elements could be manipulated by a user to adjust volume, frequency, balance, pan, trim, delay, attack time, hold time, or any other audio parameter. The control object could be inserted in an audio or virtual instrument channel of a virtual mixing board of an audio application. For example, the control object can be part of an audio effect (e.g., virtual compressor) or software instrument, which can be made available to the audio application as a "plug-in."

 

Exemplary System

 

In Apple's patent background, the context of today's new hybrid controls is that of a desktop computer. Apple stated that "If the GUI is touch sensitive, then a stylus or one or more fingers could be used to interact with the desktop." In Apple's exemplary system notes, they list Mac OS and Windows as the preferred operating systems to make the point that the primary focus is the desktop here, though not limited to the desktop.

 

When Apple lists input devices, they include "a keyboard (including a virtual keyboard), mouse, track ball, and touch-sensitive pad or display." This would support two of Apple's 2011 patent application filings illustrating their current research and work on advancing the virtual keyboard for the iMac.

 

Patent Credits and Some Closing Thoughts

 

Apple's patent application was originally filed in Q3 2010 by inventors Michael Hopkins and Robert Aron.

 

Many of Apple's patent applications over the last six months have begun to include the mention of a particular technology also applying to a touchscreen based desktop. Today's patent places direct emphasis on this throughout the patent application's verbiage and even claims. In patent claim number 8, for example, Apple states that "The method of claim 1, where augmenting the control with a slider element further comprises: detecting when a pointer, stylus or finger is at least partially touching the control object; and augmenting the knob element to provide a visual cue that the control object has been selected." If it's in Apple's patent claims, then its importance is seriously amplified. There are additional patent quotes below in our report under "3 New Apple Patents."

 

Apple's patent application for an iMac Touch caused quite the stir in the tech sector back in August 2010, and today's patent adds a little fuel to the fire that this could one day be a reality. In June 2010 we posted a special report titled "The Next OS Revolution Countdown Begins" which discussed how the user interface was likely to change over the coming years. In 2011, AMD published a PDF which discussed the next "Natural User Interface" Revolution. In October 2011, Apple introduced Siri for iPhone 4S, and today you could read about Microsoft's Kinnect for Windows.

 

It's evident that the winds of change for the next generation interface is well underway. So the concept of an iMac presenting consumers with optional UI capabilities, such as combining Multi-Touch with a Siri-like capabilities, isn't unreasonable in the least. I'm sure that Apple and the entire industry understand change is coming and they're all scrambling to be first and the best. Apple has a huge lead in this arena, yet only time will tell how this will all plays out.  

 

Apple Should Crush or Acquire their Competition

 

In light of the news that Jeff Han's Perceptive Pixel will be introducing a 27" Slanted Touch Display through vendors sometime this year, Apple should expedite the iMac Touch for key markets like advertising, drafting and more.

 

In this fantastic video describing Perceptive Pixel's, Jeff Han's admits that their display is like the iPhone and iPad's but larger. Later in this video, the price of $10,000 was mentioned for this 27" touchscreen display – though the price will come down once it hits volume, states Han. So with Apple's obvious advantages, what are they waiting for? It's time to crush their upcoming competitors before they make inroads in this emerging large touchscreen market – or buy them out if it would expedite the iMac Touch.

 

  

 

The video really shows us how this type of display could aid professional's productivity. The fluidity and speed of calling up files, moving them, arranging them, sizing them and more, just can't be matched by the aging mouse. I think that Apple could also make this type of display rock for the iMac – and in fact, three new patents prove that out. Note: we updated our report at 1 PM this afternoon with this next section about revolutionizing the desktop due to three new patent applications coming to light today. They collectively pack quite the punch.    

 

Three New Apple Patents Support Revolutionizing the Desktop

 

Interestingly, three additional patent applications from Apple today supports the same kind of thinking that was presented in the noted video above. That next generational speed and fluidity delivered to a desktop computer is addressed clearly as follows:

 

Apple's patent application 20120030569 states that "existing methods for reordering the front-to-back positions of user interface objects are cumbersome and inefficient. For example, using a sequence of mouse-based inputs to select one or more user interface objects and reorder the selected user interface objects is tedious and creates a significant cognitive burden on a user. In addition, existing methods take longer than necessary, thereby wasting energy."  

 

Apple's patent application 20120030568 states that "existing methods for copying and/or moving user interface objects between different content regions are cumbersome and inefficient. For example, copying and/or moving a user interface object from one slide to another slide in a presentation application is tedious and creates a significant cognitive burden on a user. In addition, existing methods take longer than necessary, thereby wasting energy."

 

Apple's patent application 20120030624 states that "existing methods for displaying menus are cumbersome and inefficient. For example, existing keyboard and mouse based methods for displaying menus often use multiple steps to display various menus and sub-menus, which are tedious and create a significant cognitive burden on a user. In addition, existing menu display methods take longer than necessary, thereby wasting energy."

 

 

Apple's images from today's patents noted above discuss introducing floating menus that could be tilted and the image on the right is very much like the desktop app in the video with your folders and images being scrollable down the right side.

 

Clearly Apple is now framing the argument so as to justify a future iMac Touch. The old notion of such a device being some kind of burden due to the Gorilla Arm affect no longer holds water. In context with today's video, an iMac Touch desktop that could fold down as Apple's iMac Touch patent supports, is actually the more natural and superior design for many. By being flexible, an iMac Touch would accommodate both the traditional workplace while addressing the needs of certain professional markets who could quickly take advantage of a large touch display.  Apple has the scale and the touchscreen educated customer base to make the iMac Touch a scorching winner out of the gate.

 

Apple has the advantage here with the many apps that they already have for iOS. I think that a user being able to call up iTunes, Safari, iPhoto, keynote and host of other apps into a large iMac Touch would be a blast to manipulate on screen so that the interface works fluidly with the user rather than a traditional desktop does today. Here's a video of an expert working fluidly with Keynote on an iPad. Think of how much fun that would be to work that fast on your iMac. I definitely think that this is a plus gain for the future. I think that it's just more natural. What are thoughts on this?  

 

Other Noteworthy Patent Applications Published Today

 

One

 

In May 2011Patently Apple was first to publish a report on Apple's patent covering iTunes coming to iCloud prior to iCloud being introduced by Steve Jobs in June. In that same year we posted several related reports: "Apple in the Sky with Diamonds: A Cloud Based Safe Deposit Box," another on iCloud Printing Services and yet another titled " Apple's iCloud Patent Emerges Pointing to 1TB Storage Capacity. And to top it off, we posted an in-depth report on iCloud titled " Apple's iCloud: Winds of Change."

 

Today, the USPTO published two patent applications related to iCloud which discuss the "Personal Cloud." The patent figures noted in the patent were basically the same as covered in our report covering storage capacity for iCloud. See the patent figure below. The bulk of the patent is the same, except for a number of passages concerning the Personal Cloud such as this one:

 

"In some cases, a personal cloud can dynamically adjust the distribution of services in devices based on other criteria. For example, a personal cloud can select to assign a particular service to a device based on geographic location relative to end user needs, device failure, backup or replication considerations, energy saving, growth or device capacity, expected availability of device resources, or combinations of these. This approach can allow the personal cloud to seamlessly and non-disruptively move data and services while maintaining the availability of the personal cloud services and allowing individual users to arbitrarily remove device resources from the personal cloud."

 

For those interested in every minute detail of these patents could check out patent 20120030343 Titled Dynamic Migration within a Network Storage System, as well as patent 20120030318  titled "Setup and Configuration of a Network Storage System. Here are temporary links that are valid for about 24-48 hours: One, Two. The mystery is out the bag now that iCloud is out in the market.   

 

Two

 

Apple also filed an update to an original sporting related patent which we reported on in 2009 under the title "Apple Patent Rocks the Gym." Today's patent application 20120028761 titled "Interfacing Portable Media Devices and Sports Equipment" covers the very same bases of their 2009 patent application did. It seems that another site is selling this as a new idea. It's not.  

 

Three

Another patent application 20120030623 published today covers Apple's iOS folders that could contain such items as applications, digital content, electronic documents, and/or sub-folders. You could read about creating folders for iOS devices here. Below is a patent application graphic from today's published application.  

 

 

Four

And Finally, Apple has another follow-up to their original patent application on coded magnets under application 20120028480. The patent application generally relates to programmable magnetic devices, and more particularly to multi-part devices that may be joined or separated through programmable magnets.

 

Notice: Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. Apple's patent applications have provided the Mac community with a clear heads-up on some of Apple's greatest product trends including the iPod, iPhone, iPad, iOS cameras, LED displays, iCloud services for iTunes and more. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.  

 

 

 

Here are a Few Great Sites covering our Original Report

 

MacSurfer, Frank Gillett - Forrester Research Twitter, Facebook, Cupertino Patch, University of Tennessee MacVolPlace, Real Clear Technology, StockTwits, T3, Twitter, Apple Investor News, Google Reader, Macnews, iPhone World Canada, MarketWatch, MacTechNews Germany, WebProNews, Gizmodo, Gizmodo Spain, iPad Creative, MacDailyNews, BGR, BGR Germany, Techmeme, TechRadar, Ubergizmo, Digital Spy, MacDirectory, Redmond Pie, De-Bug Germany, Synthtopia, Technorati, Huffington Post, and more.