Future Multi-Touch Feature: The Visual Expander

Apple was granted a strategically important multitouch patent back in July 2010 regarding a new multi-touch feature identified as a "Visual Expander." The funny thing is that Apple resubmitted the patent to the USPTO for some unknown reason. While it may have been to simply make a last minute tweak, the point remains that the patent is considered to be very active. And in light of Apple's announcement about a "Back to the Mac" event – anything about a future OS X feature becomes of interest. Whether this feature is for a future iteration of iOS or OS X is unknown at this time. For now, we revisit the advantages of Apple's future "Visual Expander" feature:

As you know, touch screens offer ease and versatility of operation and allow a user to make selections and move a cursor by simply touching the display screen via a finger (or stylus). Apple's granted patent states that "While touchscreens generally work well, they are difficult to use when features such as buttons, web page links, or UI controls presented on the touch screen display are too small for finger activation, i.e., the finger is too large relative to the button, link or UI controls. Furthermore, the features typically don't provide the user with any feedback indicating that a finger is located on the feature. This may make it difficult to determine what feature is being selected. As a result, the user may incorrectly select a feature. For example, in web browsing, users have no control over the size of the link or button presented by a website."

Apple's solution is to introduce us to a major new multitouch feature called the Visual Expander. Isn't that just an onscreen magnifier, you might ask. No. Apple explains the difference between the two in this way: "The virtual magnifying glass magnifies the GUI in the area of the magnifying glass, i.e. similarly to moving a magnifying glass over a printed piece of paper. The magnifying glass allows the user to traverse through the GUI so that the user could read smaller text. While virtual magnifying glasses work well, they are limited. For example, they typically don't allow features to be manipulated or selected inside the magnified area. Furthermore, they may not allow text editing therein."

Apple's granted patent notes that Apple's OS X is designed to magnify the dock including the icons when the cursor is moved over the docks icons. Yet the patent notes the limitation of this feature by stating that "While this works well, the feature has no control over the content presented on the remainder of the screen, i.e., the remaining portions of the screen do not magnify when the cursor is positioned thereover. Furthermore, this particular feature only works on the main system page. It does not work in programs or applications or even web pages."

 The Expanded State

Apple's granted patent FIGS. 9 and 10 illustrated below are diagrams showing GUI 400 in an unexpanded and expanded state. FIG. 9 is a side view and FIG. 10 is a top view. As shown, the expanded portion 402 is enlarged, magnified and raised (e.g., pushed up) relative to the remaining portions 404 of the graphical information. Although the expanded portion appears to be out of plane relative to the remaining portions, it should be noted that the expanded portion is actually in plane with the remaining portions. Visual techniques are used to make it appear as if it is raised even though it is displayed in the same plane.

In the illustrated embodiment, the expanded portion includes a plateau region 406 and a transition region 408. The plateau region is configured to display the target area 410 (noted as the yellow circle) in a larger and magnified state – which roughly translates to a magnification of 3 times that of the target area 410. This will allow most adults, especially those with larger fingers, the ability to use touch areas and controls such as hyperlinks with ease, unlike some controls on the iPhone today.

Yet until we see this feature, it's difficult to say whether or not Apple is referring to their patent pending high end 3D OS interface or not. There's a video presented in Patently Apple's report on this proposed 3D OS that clearly demonstrates how the eye could be fooled into perceiving depth that is limited to the display. This appears to be how Apple is describing their new Visual Expander feature.

Magnifying and Using Hyperlinks 

Note: Here's a quick tip to help you understand the following graphic. A portion of the patent figure 12M shown below is actually jetting out of the graphic as opposed to being dragged, as the graphic first appears at a glance.

Apple's patent describes FIG. 12M as follows: "the finger 514 is moved from the title to a link positioned within the window 520. Similar to the buttons and title, the portions of the link in the plateau are fully enlarged and magnified while the portions in the transition region are distorted and further the portions outside the expansion are of normal size and magnification. As shown in FIG. 12N, the user exerts a greater pressure on the touchscreen while the link is in its expanded form. This increased pressure is recognized as a touch event, and the element associated with link is launched or opened (although not shown)."

Magnifying Specific Zones

As shown in FIG. 13C, after the finger 514 dwells for some time over the heading, the heading is expanded. In this illustration, the heading is presented in an enlarged and magnified state while the remaining portions are not. As shown in FIG. 13D, when the finger 514 maintains contact with the touchscreen and is moved over a different GUI object as for example the field of the window, the field is expanded. In this illustration, the field is presented in an enlarged and magnified state while the remaining portions are not (including the heading).

Note: In the magnifying zones example illustrated above, the horizontal bar noted as "window" is actually jetting out in FIG. 13C. The same goes for the field of the window jetting out in FIG. 13D. With a 2D interface, you can't really see this without knowledge of what is actually being conveyed in the patent.

Magnifying Page Elements or Text

As shown in FIG. 12I, as the finger moves away from the buttons, it moves over an inside edge of the window 520 thereby causing the inside edge to be expanded as well as text in FIG. 12J.

Apple credits Peter Kennedy as the sole inventor of Patent Application 20100259500 which was resubmitted in late June of this year. The original filing was actually made in Q3 2004 and published in July 2010.

Apple Event: Back to the Mac

Yesterday Apple announced their "Back to the Mac" event that is scheduled for October 20, 2010. The logo that is shown on their invitation is rather interesting considering that it appears to be representing a likened 3D image that could very well be hinting at a future 3D version of OS X.

Apple has a number of 3D related patents that would support such a move. Interestingly, Gizmodo added a graphic to their report yesterday that is noted above along with an unknown quote supposedly from Steve Jobs that states: We've had tremendous, unbelievable success with multi-touch. And that's why starting next year, we're bringing it to the Mac." If true, it would validate Apple's recently published iMac Touch patent that Macites globally applauded back in August.

I for one will greatly appreciate clarification of their operating system roadmap. I'm close to pulling the trigger on a new iMac but have one eye looking to 2011 for two reasons: 1) Intel's Sandy Bridge or AMD's Fusion, and 2) an advanced next generation OS.

The two noted issues are actually interrelated. We discussed that scenario in two of our reports titled " The Next OS Revolution Countdown Begins" and "Intel's CES Keynote 2010, Apple and iLife 3D." In fact, a third report of ours touched on the subject regarding the advent of Heterogenous Computing which takes advantage of Apple's OpenCL at the chip level.

This is what I'm hoping that Apple will clarify to some degree next week. Is Apple bringing multi-touch to the iMac? Will 3D be coming to OS X? Will we need to have a Mac with one of one of the next generation architectures noted above to be able run these features? Will Apple be shifting to AMD? With Otellini taking another crack at Apple, anything is possible.

That's what will make next week's big event so much fun. We may finally get some meaty answers to some really important questions. And today's patent only reminds us all once again that Apple is continually making huge strides in the advancement of multi-touch features that may one day land on our Macs.

At the end of the day, what do you think will unfold next week? You've almost had a full day to think about it – so what's on your mind today? Send in your predictions in our comment area below. We'd love to hear what you're thinking.

Cheers!

Notice: Patently Apple presents only a brief summary of patents with associated graphic(s) for journalistic news purposes as each such patent application and/or Granted Patent is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application and/or Issued Patent should be read in its entirety for further details. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.