On August 2, 2007, the US Patent & Trademark Office published Apple's patent application titled Multi-touch Gesture Dictionary. At first glance, Apple's patent could be taken as quite intimidating. I mean, who on earth really wants to learn an entirely new gesturing language just to be able to use your iPhone. Not many. However, for anyone who has ever learned the guitar, you know that it too looked insanely difficult at first. Yet with a little practice and some mean jamming with your friends, playing chords became second nature. Whether that proves to be the same for Apple's sophisticated gesturing language for the iPhone is another matter. I also know some mean playing numeric-pad accountants, and at the end of the day, I couldn't care less in learning that skill. Which way will the wind blow on this one? I don't know. Will it be viewed as cool to know the new Apple iPhone chord gesturing language or just a Mac cultists dream. Only time will tell on that one. For now however, Apple's patent is an interesting read.
Apple's Patent Background
Many attempts have been made over the years to improve the way users interact with computers. In the beginning, cards or tapes with punched holes were used for user input. Punch cards gave way to terminals with alphanumeric keyboards and text displays, which evolved into the modern keyboard, mouse, and graphical-display based graphical user interfaces. Many expect that the use of multi-finger, touch-sensitive user interfaces ("multi-touch interfaces", such as those described in the references incorporated above, will become widely adopted for interacting with computers and other electronic devices, allowing computer input to become even more straightforward and intuitive.
Users of these multi-touch interfaces may make use of hand and finger gestures to interact with their computers in ways that a conventional mouse and keyboard cannot easily achieve. A multi-touch gesture can be as simple as using one or two fingers to trace out a particular trajectory or pattern - or as intricate as using all the fingers of both hands in a complex sequence of movements reminiscent of American Sign Language. Each motion of hands and fingers, whether complex or not, conveys a specific meaning or action that is acted upon by the computer or electronic device at the behest of the user. The number of multi-touch gestures can be quite large - because of the wide range of possible motions by fingers and hands. It is conceivable that an entirely new gesture language might evolve that would allow users to convey complex meaning and commands to computers and electronic devices by moving their hands and fingers in particular patterns.
A Multi-touch Gesturing Language Dictionary
To take full advantage of a multi-touch gesture language, users will need to learn and/or remember the meaning of numerous gestures. One learning or trying to remember the meaning of words in a verbal language often makes use of a dictionary, essentially a list of words and their associated meanings. In an analogous manner, one learning or trying to remember the meaning of gestures could consult a gesture dictionary, e.g., a list of gestures and their associated meanings.
Although it is possible to learn and/or look up the meanings of gestures using a gesture dictionary formed around verbal descriptions, this may not be efficient for at least three reasons. First, the gesture itself may not be known to the user. Second, the meaning of the gesture may change as a function of the context in which it is performed. Third, the index of possible gestures may not be easily describable in words, thus making searching a verbal index cumbersome.
Furthermore, learning a multi-touch gesture language may be facilitated by having a gesture dictionary that provides some type of demonstration of the expected hand and finger motion. Similarly, remembering a previously learned gesture's meaning may also be benefited by having some way to easily access the meaning associated with the particular gesture.
Therefore, disclosed herein is a gesture dictionary that facilitates the learning and retention of the meanings or definitions of gestures that make up a multi-touch gesture language by providing demonstration of expected hand and finger motions. The gesture dictionary disclosed herein further allows looking up or accessing the meaning of gestures in a quick and easy manner that does not depend on verbal indexing.
Sophisticated Gestures: Recognized Chords
According to Apple, a chord can include a set of fingers from either or both hands that initially contact a multi-touch surface prior to motion on the multi-touch surface. In many multi-touch systems the chord may uniquely specify a set of gestures that belong to the combination of fingers and orientations making up the chord.
Each of a user's hands can execute twenty-five or more chords. For example, five fingers that can be independently raised or lowered give rise to twenty-five combinations. Additional chords may be distinguished by whether only the fingertips are in contact with the surface or whether the length of the finger is flattened against the surface. Further chords may be distinguished based on whether the fingertips are placed on the surface close together or spread apart.
Many chords can have at least thirteen different motions associated with them. For example, a two-finger chord (for example, the index and middle fingers) could have specific meaning or action assigned to the lateral motions that include rotation, translation, and scaling. Rotation (clockwise and counter-clockwise) of the two-finger chord gives rise to two unique meanings or actions. Translation (left, right, up, down, and four diagonals) gives rise to at least eight unique meanings or actions. Scaling (contraction or expansion) also gives rise to two meanings or actions. The vertical motion of a chord may comprise lifting the fingers of the chord off the multi-touch surface almost immediately after they had touched down, (e.g., tapping the multi-touch surface with the chord) or multiple taps, etc.
With each hand able to execute twenty-five or more chords, and with each chord having thirteen or more motions associated therewith, there may be over three hundred possible gestures for each hand. Many more gestures are possible if both hands are used together. This gives rise to the gesture language referenced above.
One approach to creating a gesture dictionary indexes the dictionary using the chords, much as a textual dictionary uses the alphabet. For example, just as there may be a particular number of words that start with a particular letter, so there may be a particular number of gestures that start with a particular chord. These gestures may be presented to a user in a way that facilitates rapid assimilation by the user. For example, template 100 for a combination graphical and textual dictionary entry for a given chord is illustrated in FIG. 1 above.
Thumb & 1 Finger Chords
Apple's patent FIG. 2 illustrates dictionary entry 200 for commands that may be associated with gestures starting with a "thumb and one finger" chord. Specifically, a thumb and one finger chord followed by upward motion 202 can correspond to an undo command. Similarly, a thumb and one finger chord followed by downward motion 208 can correspond to a redo command. It should be noted that it may aid users' attempts to learn and remember gestures for complementary commands to have complementary motions associated with them in this manner. Other commands that can correspond to the thumb and one finger chord include tab (associated with rightward motion 206), back tab (associated with leftward motion 204), copy (associated with tap 205), cut (associated with contraction 213, e.g., a pinching motion), and paste (associated with expansion 212, e.g., the reverse of a pinching motion).
As seen in FIG. 2, certain motions of the thumb and one finger chord do not have a command associated with them, e.g., upward left motion 201, upward right motion 203, downward left motion 207, downward right motion 209, counter-clockwise rotation 210, and counterclockwise rotation 211.
Thumb & 2 Finger Chords
Apple's patent FIG. 3 illustrates exemplary dictionary entry 300 for commands that may be associated with gestures starting with a "thumb and two finger" chord.
In this example, the standard thumb and two finger chord followed by any translational motion (i.e., translation upward and to the right 301, translation upward 302, translation upward and to the left 303, translation to the left 304, translation to the right 306, translation downward and to the left 307, translation downward 308, and translation downward and to the right 309) may be associated with a dragging operation as might be accomplished in conventional graphical user interface ("GUI") systems by holding a mouse button while moving the mouse. Tap 305 of the thumb and two finger chords may correspond to a right click command. Counter clockwise rotation 310 or clockwise rotation 311 following a thumb and two finger chord may correspond to group and ungroup commands, respectively. Expansion 312 and contraction 313 of the thumb and two finger chords may correspond to replace and find commands, respectively.
Thumb & 3 Finger Chords
Thumb & 3 Finger Chords
Apple's patent FIG. 4 illustrates dictionary entry 400 for commands that may be associated with gestures starting with a standard "thumb and three finger" chord, as distinguished from a spread "thumb and three finger" chord, described below in connection with FIG. 5. In the given example, the standard thumb and three finger chord followed by upward motion 402 can correspond to a parent directory command, i.e., moving up a directory level in a file browser or similar application. A standard thumb and three finger chord followed by downward motion 408 can correspond to a reload command, as would be used in a web browser application, for example. Continuing with command that might be associated with browser-type applications, left translation 404 or right translation 406 may correspond to back and forward commands common in browser applications. Other commands that can correspond to the thumb and three finger chord include open and close, corresponding to counter-clockwise rotation 410 and clockwise rotation 411, and new and save, corresponding to expansion 412 and contraction 413.
Spread Thumb & 3 Finger Chords
Apple's patent FIG. 5 illustrates dictionary entry 500 for commands that may be associated with gestures starting with a spread "thumb and three finger" chord. In brief, a spread chord may be executed with the fingers making up the chord (in this case a thumb and three fingers, e.g., the index, middle, and ring fingers) substantially spread apart. Conversely, a standard chord may be executed with the fingers making up the chord in a neutral, relaxed posture. An Alternative Gesture Dictionary for Beginners As an alternative or supplement to the dedicated gesture dictionary application described above, it may be desirable for a user to access the gesture dictionary quickly from a program application being used to perform a particular task, i.e., not a dedicated dictionary application. For example, a user may desire to perform a particular command in the program application, but may not remember the gesture associated with the command or may remember the chord but not the motion making up the gesture. Thus, another embodiment of the gesture dictionary may take the form of a background program that presents a dictionary entry associated with a chord if that chord is presented without any of the motions associated with that chord being performed within a predetermined time delay. A dictionary entry may also be presented if a gesture is performed that does not have a meaning associated with it (e.g., the thumb and one finger chord followed by rotation discussed above) or if a gesture is performed that, as determined by program logic, does not make sense in the particular context.
Apple's patent FIG. 5 illustrates dictionary entry 500 for commands that may be associated with gestures starting with a spread "thumb and three finger" chord. In brief, a spread chord may be executed with the fingers making up the chord (in this case a thumb and three fingers, e.g., the index, middle, and ring fingers) substantially spread apart. Conversely, a standard chord may be executed with the fingers making up the chord in a neutral, relaxed posture.
An Alternative Gesture Dictionary for Beginners
As an alternative or supplement to the dedicated gesture dictionary application described above, it may be desirable for a user to access the gesture dictionary quickly from a program application being used to perform a particular task, i.e., not a dedicated dictionary application. For example, a user may desire to perform a particular command in the program application, but may not remember the gesture associated with the command or may remember the chord but not the motion making up the gesture. Thus, another embodiment of the gesture dictionary may take the form of a background program that presents a dictionary entry associated with a chord if that chord is presented without any of the motions associated with that chord being performed within a predetermined time delay. A dictionary entry may also be presented if a gesture is performed that does not have a meaning associated with it (e.g., the thumb and one finger chord followed by rotation discussed above) or if a gesture is performed that, as determined by program logic, does not make sense in the particular context.
The time delay may prevent a gesture dictionary entry from being presented to the user every time a gesture is performed, which could be an annoyance or distraction. However, in some modes of operation this time delay could be omitted or substantially shortened. For example, it may be beneficial to a beginning multi-touch gesture user to have the dictionary entries displayed after every chord as a learning reinforcement mechanism.
The flow chart in FIG. 6 shows the steps of accessing the gesture dictionary from another application as described above.
The multi-touch system may continuously monitor the multi-touch surface looking for the presence of a chord (601). When a chord is detected the system may monitor the positions of the fingers making up the chord looking for lateral or vertical motion on the surface (602). Each time the system checks for motion and there is none, a counter may be incremented before the system checks for motion again (604). If motion is detected before the counter reaches a predetermined value N then the combination of chord and motion (i.e., the gesture) may be processed and the current meaning or action associated with the gesture may be executed (603). If, however, motion is not detected by the time the counter reaches the value N (605), then the system may open the gesture dictionary to the entry corresponding to the chord being presented by the user (606). The counter thus implements the time delay function discussed above.
Once the dictionary application is opened (606), the system may determine whether a motion has started (607). If so, the application may temporarily display feedback associated with the gesture (608), e.g., highlight a motion icon associated with the gesture, and then process the gesture, e.g., execute the meaning or action associated with the gesture (603). If motion has not yet started, the system may check to see if the chord has lifted off. If so, the dictionary display may be closed - (see 610). If no liftoff is detected, the system may continue to check for motion and/or liftoff (607, 609) until one or the other is detected and then process the result accordingly.
Exemplary Gesture Dictionary UI
Apple's patent FIG. 7 illustrates an exemplary user interface display for a gesture dictionary application that may be used for assigning meanings to gestures. As in the examples discussed above, the display for a particular chord may include plurality of motion icons 701. Particular motion icon 702 corresponding to the gesture currently being edited may be highlighted or otherwise indicated in some way to provide indication to the user of the gesture currently being edited. Dialog box 703 may show, for example, whether the meaning associated with the particular motion (gesture) is the default or a custom mapping. A plurality of GUI buttons 710 may also be provided so that a user can indicate the assignment of a meaning to a gesture is completed ("Done"), cancel the assignment of a meaning to a gesture ("Cancel"), restore the default meanings, or clear all of the custom meanings.
Possible Multi-Touch Devices
The devices that could eventually use multi-touch gesturing, according to the patent, are illustrated below in FIG. 10. Apple lists them as follows: "desktop computers 1001, notebook computers 1002, tablet computers 1003, handheld computers 1004, personal digital assistants 1005, media players 1006, mobile telephones 1007, and the like. Additionally, the computer may be a combination of these types, for example, a device that is a combination of a personal digital assistant, media player, and mobile telephone."
Apple credits John Greer Elias, Wayne Carl Westerman and Myra Mary Haggerty as the inventors of patent 20070177803.
NOTICE: Patently Apple presents only a brief summary of patents with associated graphic(s) for journalistic news purposes as each such patent application and/or grant is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application and/or grant should be read in its entirety for further details. For additional information on today's patent(s), simply feed the individual patent number(s) noted above into this search engine.
Update Note: The segment/paragraph titled "Possible Multi-Touch Devices" was added to this original report on August 22, 2009. The content in this segment under quotes is directly from the patent as identified in this report. The extra graphic with displayed devices could be found in Apple's patent.