Apple Patent Reveals Voice Commands Coming to iPod, iPhone & Apple TV
Apple Reveals a Highly Advanced Battery Charging System Patent

Apple Reveals New Multi-Dimensional GUI

A new multidimentional desktop in development cover 

In November, the UPSTO Commissioner for Patents told the EE Times that his office was still falling behind and the backlog was growing. That understatement was clearly evidenced today when the UPSTO published between 65-70 Apple patents in a single day. While there were several technological themes that emerged within this allotment, such as back-up and programming, the number one theme of the day dealt with a new multi-dimensional GUI from Apple. "As the capabilities of processing devices progress" the patent reads, "so do the demands on the graphical user interface to convey information to the users in an intuitive manner." Apple defines multi-dimensional as an interface capable of displaying 2.5D or 3D imagery. Apple introduces us to the Newtonian physics model that could be utilized in this new GUI and presents us with several patent figures that illustrate coming features and/or effects. One of these cool effects includes the ability to wrap open windows around multiple surfaces. There are approximately 12 patents relating to this proposed GUI and this report simply presents you with a few highlights.   


Multi-Dimensional Desktop & Application Environment


Apple's patent FIG. 2 shown below is a block diagram of an example user interface architecture (200). The user interface architecture includes a user interface (UI) engine (202) that provides the user access to the various system objects (204) and conveys information about the system to the user.


Multi-dimensional Desktop & Application Environment 
Upon execution, the UI engine can cause the graphics device to generate a graphical user interface on an output device, such as a display device. In one implementation, the graphical user interface can include a multidimensional desktop (210) and a multidimensional application environment (212). In an implementation, the multidimensional desktop and the multidimensional application environment include x-, y- and z-axis aspects, e.g., a height, width and depth aspect. The x-, y- and z-axis aspects may define a three-dimensional environment, e.g., a "3D" or "2.5D" environment that includes a z-axis, e.g., depth, aspect.


In an implementation, the multidimensional desktop can include use interface elements, such as visualization objects (220), a visualization object receptacle (222), and stack items (224). In some implementations, the visualization objects, the visualization object receptacle and the stack items can be presented in a pseudo-three dimensional (i.e., "2.5D") or a three-dimensional environment as graphical objects having a depth aspect.


A visualization object can, for example, be a visual representation of a system object. In some implementations, the visualization objects are icons. Other visualization objects can also be used, e.g., alert notification windows, menu command bars, windows, or other visual representations of system objects.


In an implementation, the multidimensional application environment can include an application environment distributed along a depth aspect. For example, a content frame, e.g., an application window, can be presented on a first surface, and control elements, e.g., toolbar commands, can be presented on a second surface.


The User Interface Engine Architecture


Apple's patent FIG. 5 shown below is that of a block diagram of an example user interface engine architecture (500). The UI engine can, for example, include an interaction and visualization model engine (502), a physics engine (504), and a context engine (506). Other engines can also be included.


The User Interface Engine Architecture 
In one implementation, the physics engine can apply a physics aspect, such as Newtonian physics models based on mass, velocity, etc., to the visual representations of system objects, such as icons. In an implementation, the icons can be modeled as rigid bodies or non-rigid bodies. For example, placing an icon on a surface next to adjacent icons can cause the adjacent icons to shift positions in response to a simulated disturbance from the icon placement. In one implementation, icon magnetism can be selectively enabled or disabled by the user. In one implementation, icons return to their initial positions upon a disabling of the magnetism aspect. In another implementation, a magnet icon can have a magnetism aspect selected by the user, e.g., a magnetism with respect to a word processing application, or a magnetism with respect to two or more applications, or a magnetism with respect to the last time a document was accessed, e.g., within the last two days, etc.


Apple's patent FIG. 6 noted above is a block diagram of example system layers (600) that can be utilized to implement the systems and methods described herein. Other system layer implementations, however, can also be used.


In an implementation, a user interface engine, such as the UI engine or another UI engine capable of generating a three-dimensional user interface environment operates at an application level (602) and implements graphical functions and features available through an application program interface (API) layer (604). Example graphical functions and features include graphical processing, supported by a graphics API, image processing, support by an imaging API, and video processing, supported by a video API.


The API layer, in turn, interfaces with a graphics library layer (606). The graphics library layer can, for example, be implemented as a software interface to graphics hardware, such as an implementation of the OpenGL specification.


Examples of Multi-Dimensional Desktop Environments


The following patent figures illustrating multi-dimensional environments are extremely detailed and can't be covered adequately an overview type of report such as this. Patently Apple strongly suggests that those who are interested in Apple's extreme detailing review the original patent (20080307360) verbiage and match it up with our illustrations below to assist your understanding.


Apple's patent FIG. 7, noted below, is a block diagram (700) of an example multidimensional desktop environment. In the example implementation, the multidimensional desktop environment includes a back surface (702) axially disposed, e.g., along the z-axis, from a viewing surface (704). In one implementation, the back surface can, for example, be a two-dimensional desktop environment, including one or more menus (701 and 703). In one implementation, the viewing surface can be defined by the entire image on a display device, e.g., a "front pane." One or more side surfaces, such as side surfaces (706, 708, 710 and 712), are extended from the back surface to the viewing surface. An icon (714) is generated on one or more of the side surfaces (706). Although only one visualization object receptacle is shown, addition icon receptacles can also be displayed, e.g., along the side surface (708).


In one implementation, a reflection region (716) can be generated on the side surface, e.g., the "floor." In an implementation, a reflection of the back surface and of graphical items placed on the reflection region can be generated, e.g., shapes (760 and 762) generate reflections (761 and 763) in the reflection region.


In one implementation, a motion model is dependent on a selected surface aspect. For example, an equable texture, such as an image of a hardwood floor or a polished metallic surface, can be associated with a rigid-body Newtonian physics model; conversely, a visible grid aspect, or a raised texture, such as an image of a carpet, pebbles, etc., can be associated with a grid snap. In another implementation, the motion mode and textures can be selected independently.


Multidimensional desktop reflections, motion V3 
Apple's patent FIG. 11 shown below is a block diagram of another example multidimensional desktop environment which includes a back surface.  In another implementation, the lighting aspect can generate an illumination effect from the window thumbnail 1170 onto one or more surfaces. For example, the illumination effect can comprise a simulated sunbeam emanating from one window 1170 - with 1172 being a reflection or shadow of window 1170. In one implementation, the illumination effect can change according to local environmental states, e.g., the sunbeam can track across the surfaces according to a local time; the intensity of the sunbeam can be modulated according to the local time and local weather conditions that are received over the network 118, e.g., high intensity for sunny days, low intensity for overcast days and during the early evening, and/or being eliminated after a local sunset time and generated after a local sunrise time.


Multi-dimensional Desktop with simulated Sunbean 
Apple's patent FIG. 14, noted below is a block diagram of another example multidimensional desktop environment which also facilitates a multidimensional application environment. For example, an application content frame (1410), e.g., a window displaying editable data, can be displayed on the back surface (1102), and one or more application control elements can be displayed on one or more side surfaces. For example, a three-dimensional function icon arrangement (1420) can be displayed on the surface 1108, and menu items 1430 can be displayed on the surface (1112).


Multidimensional desktop - multidimensional application enviroment 

The three-dimensional function icon arrangement can, for example, include three-dimensional function icons (1422, 1424, 1426 and 1428). In one implementation, each three-dimensional function icons include a function command on each surface, and each three-dimensional function icon. can be rotated, positioned, and manipulated through the use of an input device, such as a mouse.


Windows Could be Dragged or Displaced on one or more Surfaces


Apple's patent FIG. 15 noted below is a block diagram of another example multidimensional desktop environment which illustrates how windows could be dragged or displaced across one or more surfaces. That could be a very cool feature.


Multidimensional desktop, windows can be dragged unto multi surfaces 

Other Miscellaneous Examples


Miscellaneous - multidimensional examples 

Multidimensional Desktop Environment Geometries


Apple's patent FIG. 16A shown below is a block diagram of another example multidimensional desktop environment which illustrates an arcuate back surface (1602 - curved like a bow) that is axially disposed, e.g., along the z-axis, from a viewing surface 1604.


Multi-dimensional - with arcuate back surface 

Other multidimensional desktop environment geometries can also be used. For example, in one implementation, the multidimensional desktop environment can conform to a tetrahedron-shaped environment (like a pyramid) in which a front surface of the tetrahedron defines a viewing surface, and the remaining three surfaces define a left surface, a bottom surface, and a side surface.


Apple credits Imran Chaudhri, John Louch, Christopher Hynes, Timothy Bumgarner and Eric Peyton as the inventors of 20080307360, originally filed in Q2 2007.


The Mirror Effect: An Exemplary Physical Arrangement


Multidimensional desktop feature - reflection content 
In a secondary related patent (20080307366), Apple presents us with patent FIG. 5B (as shown above) which illustrates a block diagram of an exemplary physical arrangement which the reflection content surface 508 mimics. In an implementation, the reflection content surface can appear to a viewer observing along viewing angle 522 as a mirror at right angle to the visualization objects 516 and 518, and containing a reflection of the visual content of the back surface 502 and the application interfaces 504 and 506. The reflection surface 508 is not limited to being placed at the bottom of a multidimensional desktop environment 500 (i.e., to reflect upward), and can be placed anywhere in the multidimensional desktop environment to reflect content in any orientation. In an implementation, scaling of the reflected content can be applied so as to simulate a curved reflecting surface.


Apple credits Michael James Paquette, Bas Ording, John Louch and Imran Chaudhri as the inventors of patent 20080307366, originally filled in Q2 2007.


Other important multi-dimensional related patents that were published today include the following: Multi-Dimensional Application Environment – 20080307351; Visualization and Interaction Models – 20080307334; Three Dimensional Viewer for Video - 20080307309 and one finally one that is titled 3D Movie Browser or Editor covered in this patent report.


Notice: Patently Apple presents only a brief summary of patents with associated graphic(s) for journalistic news purposes as each such patent application and/or grant is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application and/or grant should be read in its entirety for further details. For additional information on any patent reviewed here today, simply feed the individual patent number noted above into this search engine


The comments to this entry are closed.