On January 12, 2012, the US Patent & Trademark Office published a patent application from Apple that reveals an exciting new 3D GUI for iOS mobile devices. The new UI will work with proximity sensor arrays and will respond to hovering gestures. The Crazy Ones in Cupertino have been working on advanced 3D GUIs for some time now. We first learned of a major 3D GUI project back in 2009 that involved using head tracking technology. Then in early 2010 we learned of Apple's first project relating to a 3D GUI for iOS devices. Later that year Apple 3D multifunctional widgets and over time revealed advanced 3D and hovering based gesturing for CAD users on an iPad. With twenty times the GPU power coming to iOS devices over the next year, Apple appears to paving the way for a new 3D GUI for mobile devices in the not-too-distant future. Update Saturday Jan 14, 2011, 2PM MST: We've added a video to the report.
Apple's Patent Background
Modern computer operating systems often provide a desktop graphical user interface ("GUI") for displaying various graphical objects. Some examples of graphical objects include windows, taskbars, docks, menus and various icons for representing documents, folders and applications. A user could interact with the desktop using a mouse, trackball, track pad or other known pointing device. If the GUI is touch sensitive, then a stylus or one or more fingers could be used to interact with the desktop. A desktop GUI could be two-dimensional ("2D") or three-dimensional ("3D").
Modern mobile devices typically include a variety of onboard sensors for sensing the orientation of the mobile device with respect to a reference coordinate frame. For example, a graphics processor on the mobile device could display a GUI in landscape mode or portrait mode based on the orientation of the mobile device. Due to the limited size of the typical display of a mobile device, a 3D GUI could be difficult to navigate using conventional means, such as a finger or stylus. For example, to view different perspectives of the 3D GUI, two hands are often needed: one hand to hold the mobile device and the other hand to manipulate the GUI into a new 3D perspective.
Apple's Wild Solution
Apple's latest invention is a wild one indeed. The invention covers a 3D display environment for mobile device that uses orientation data from one or more onboard sensors to automatically determine and display a perspective projection of the 3D display environment based on the orientation data without the user physically interacting with (e.g., touching) the display. In some implementations, the display environment could be changed based on gestures made a distance above a touch sensitive display that incorporates proximity sensor arrays.
In some implementations, a computer-implemented method is performed by one or more processors onboard a handheld mobile device. The method includes generating a 3D display environment; receiving first sensor data from one or more sensors onboard the mobile device, where the first sensor data is indicative of a first orientation of the mobile device relative to a reference coordinate frame; determining a first perspective projection of the 3D display environment based on the first sensor data; displaying the first perspective projection of the 3D display environment on a display of the mobile device; receiving second sensor data from the one or more sensors onboard the mobile device, where the second sensor data is indicative of a second orientation of the mobile device relative to the reference coordinate frame; determining a second perspective projection of the 3D display environment based on the second sensor data; and displaying the second perspective projection of the 3D display environment on the display of the mobile device.
Apple's 3D Environment for Mobile Devices
Apple's patent FIG. 1A shown below illustrates a perspective projection in a 3D display environment of a mobile device using orientation data derived from onboard sensors.
In some implementations, the mobile device includes display 102 presenting display environment 104. For example, the mobile device could include a touch-sensitive surface that could receive touch input and gestures. The display environment could be a 3D GUI having a floor (106), back wall (108), ceiling (110) and sidewalls (112a and 112b). One or more two-dimensional ("2D") or 3D objects could be presented in the display environment. In the example shown, 2D object (118) is pasted to the back wall and 3D objects (114 and 116) are resting the on floor.
Apple's proposed display environment could be a perspective projection of a 2D desktop or homepage for an operating system or application. The perspective projection could be conceptually understood to be the view from an imaginary camera viewfinder. The camera's position, orientation and field of view could control the behavior of a projection transformation or camera transform.
Shifting Camera Views
Apple states that the "camera view" of the display environment changes when a viewer rotates their mobile device about a gyro sensor axis. If no gyro sensor is available another sensor such as the magnetometer or accelerometer could be used.
In some implementations, the display environment is determined based on the current orientation of the mobile device as indicated by one or more onboard sensors. For example, the display environment could be displayed when the viewer holds their mobile device with the display directly facing them. In this orientation, the camera view is directly facing the back wall. As the user rotates their mobile device either clockwise or counterclockwise, the camera view is moved towards either side wall.
More particularly, as the user rotates their mobile device clockwise (about the X axis of rotation), the camera view moves toward sidewall 112a. Likewise, as the user rotates their mobile device counterclockwise, the camera view moves toward sidewall 112b. As the user rotates their mobile device clockwise (about the Y axis of rotation), the camera view moves toward the ceiling. As the user rotates their mobile device counterclockwise (about the Y axis), the camera view moves toward floor.
By rotating mobile device about the X and/or Y axes, the viewer is easily able to change the view of their display environment as desired to view various icons or documents as presented on the display.
Angular rotation about the X and Y axes could be measured or estimated from data provided by gyro sensors, accelerometers, magnetometers or any combination of sensor data that could provide an estimate of the orientation of the mobile device relative to a reference axis of rotation.
Apple Introduces a "Snap To" Feature
Apple's patent FIG. 1B shown below illustrates a perspective projection in a 3D display environment using orientation data derived from onboard sensors, and including a "snap to" feature for changing camera position in the display environment. Note what was on the left wall of patent figure 1A is now positioned in the center back wall of the 3D environment in patent figure 1B below. As the user rotates their device, a new view of their display environment could be seen and snapped into place so that they could interact with it.
When using the mobile device, there are physical constraints on the amount of any particular rotation. If the viewer rotates their mobile device about the X or Y axis too far, the viewer will no longer see their display environment 104.
Exemplary Process
Apple's patent FIG. 2 shown below is a flow diagram of an exemplary process 200 for determining perspective projections in a 3D display environment using orientation data derived from onboard sensors.
Multiple GPUs and a 3D Graphics Rendering Engine
In some implementations, the process could begin by generating a 3D display environment for a display of a mobile device. The 3D display environment could be displayed using one or more graphics processing units (e.g., NVIDIA GeForce 330M) and a 3D graphics rendering engine, such as the open source object-oriented graphics rendering engine ("OGRE").
Yet a development this week may prove to be the more likely GPU candidate to power Apple's proposed 3D GUI for mobile devices. At the Consumer Electronics Show in Vegas this week, Imagination Technologies presented their new PowerVR G6200 and G6400 GPU IP cores – which may be able to offer up to 20 times the performance of smartphones currently on the market today. We discussed advances coming from Imagination Technologies in our December 12, 2011 report titled "Apple Pounds out a New Round of Camera Related Patents." Being that today's 3D GUI patent relies heavily on utilizing the cameras built-into Apple's iOS devices, the developments presented in Vegas this week are very relevant.
3D Environments for a Full Range of Devices
Some examples of mobile devices that will be able to implement Apple's proposed 3D environment include but are not limited to: a handheld computer, iPod touch (a personal digital assistant), an iPhone, an iPod, iPad, network appliance, a camera, an enhanced general packet radio service (EGPRS) mobile phone, a network base station, an iPod, a navigation device, an email device, a game console, or a combination of any two or more of these data processing devices or other data processing devices.
The War over the Next Great Interface will be with us for a Decade
In October 2011, Microsoft revealed their upcoming Metro UI for portable devices and made a pitch at how it would be so much better than Apple's use of static icons. Well, just when Apple knew that Microsoft would be trying to copy iOS in some fashion, we now see Apple's revelation of how they may take a future iteration of iOS to the next level. Time will tell of course if Apple gets this 3D GUI out to market in a timely manner. But at the moment, it looks very promising indeed. In the bigger picture, the war over the next great interface for larger tablets, TVs and our vehicles are still up for grabs. But between Apple's advanced work on 3D graphical user interfaces as well as audio-centric user interfaces like Siri, it sure looks like Apple has the wind at its back.
Apple's patent applicationwas originally filed in Q3 2010 by inventor Patrick Piemonte. For more on Apple's 3D related patent applications, see our Archives.
Other Noteworthy Patent Applications Published Today
Apple has designed and invented a new power adapter with an arm mechanism that is manufactured from a single piece of material. The plug arm could include a plug operative to extend into a wall socket, an elongated plate coupled to an end of the plug such that the plug extends from a first surface of one end of the plate, and a pin coupled to the opposite end of the plate and extending from the opposite surface of the plate.
That's a very nice description from Apple, but it's still an odd power adapter to me. Yet perhaps the super geeks amongst us could make sense of this (ha!) - and let us know below in our comment section.
For more information, see patent application 20120005897.
Notice: Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. Apple's patent applications have provided the Mac community with a clear heads-up on some of Apple's greatest product trends including the iPod, iPhone, iPad, iOS cameras, LED displays, iCloud services for iTunes and more. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.
Update Saturday 2:10 PM MST: There's an interesting iOS developer who created an iApp called "Holo Toy." It involves some of the principle's laid out in Apple's patent application. The video shown below is worth viewing. It will give you an idea of a 3D environment on both an iPhone and iPad. If you skip around the video, you get to see several excellent example's of differing 3D gaming window designs including one where you could spin a globe in a 3D environment. This is something that Apple could create via the technology that they acquired from Poly9 of Quebec, Canada, back in July 2010. Although the video is primarily covering a game, the view of this 3D environment provides you with an idea of what a 3D GUI would like like in general. Of course working with traditonal apps and being able to manipulate the environment to display documents and/or a photo-wall is far more complex. But it's a great visual nonetheless.
Ben, who created this app, emailed me this afternoon and stated the following: "HoloToy uses the accelerometer, gyroscope and head tracking to alter the perspective of a 3D view - giving the illusion of a 'real' 3D space existing 'within' the device. HoloToy includes multiple interactive toys, games and even a 3D photo editor. Combining this with the proximity-sensor array for mid-air gestures sounds amazing! " Excellent Ben. Ben's Holo Toy iApp is available at the App Store. Who knows, maybe Apple will be giving Ben a call. Time will tell.
Glasses-Free 3D News Jan. 23, 2012: LG Shows Glasses-Free 3D PC Display and Notebook
Here are a Few Great Community Sites covering our Original Report
MacSurfer, Twitter, Facebook, Reddit, Apple Investor News, Google Reader, Macnews, iPhone World Canada, Tech Spy, Xponent4, The Metaverse Journal, LeiPhone China,TuexpertoMovil Spanish, AppleWeblog Spain, phoneArena, The APPera, MarketWatch, Agoracom, MacDailyNews, iPhoneItalia Italy, Blog do iPhone Brazil, MacRumors, iDevice Romania, Gizmodo, 9to5 Mac, iClarified, Gizmovil Spain, iPhoneHellas Greece, iPhone Hacks, I Love Apps China and PadGadget.
Techmeme, Computerworld, Mashable (Video Brief), Phonemantra, India (English), Gizmodo France, Gizmodo Spain, Gizmodo Germany, DragonFlyEye.Net, TICbeat Spain, Apfelzone Germany, iCreate Netherlands, iPhoneRumors Italy, Macnotes Germany, Mac Life Germany, Klamm Germany, BlogiPhone France, 33rd Square, Cultura Mexico, TechDigest, AKAScope, Redmond Pie, Movilzona Spain, Appleunity Germany, RegHardware UK, Quaeng Germany, LesMobiles France, Peter Bryer's Mobile Foresight (Works at Nokia), and more.
Note: The sites that we link to above offer you an avenue to make your comments about this report in other languages. These great community sites also provide our guests with varying takes on Apple's latest invention. Whether they're pro or con, you may find them to be interesting, fun or feisty. If you have the time, join in!
@Spangle
I wouldn't rely on how Apple presents their IMO for how they'd handle an iOS, OSX or application 3D Homepage. Apple's Instant Messaging isn't true 3D anyways. So it's a terrible example. As I've stated earlier, you're a naysayer and so this type of response was kind of expected. Yawn.
Posted by: MonkeyMo | January 16, 2012 at 05:48 AM
@MonkeyMo
Apple do build great UIs, but IMO they rely too much on skeuomorphs that are distracting and add little to the usability of the application. Most of this behaviour is 3D animation too, so I don't hold out any real hope that a complete 3D Ui done by Apple will add value - it hasn't so far.
Posted by: spangle | January 16, 2012 at 02:20 AM
Very cool video. Nice images that really get the imagination fired up. Kudos to you and Ben's little app.
Posted by: MonkeyMo | January 14, 2012 at 02:51 PM
i just want a waterproof iPhone.
Posted by: nick | January 13, 2012 at 11:53 AM
Very cool. The fight for the interface of the future (or the many interfaces of the future) is well underway. Siri will be reading your lips, and possibly your thoughts.
Posted by: 33rd Square | January 13, 2012 at 08:34 AM
@ Spangle.
With your naysaying attitude, Steve Jobs would have said, MP3 players are already out there, what can we do? What Apple did was reinvent the MP3 player and did it "right."
With your naysaying attitude, Steve Jobs would have said, we did Newton and Microsoft did tablets. It just doesn't work. What Apple did was reivent the PDA and Smartphone and did it "right."
I'm so tired of naysaying dopes who are supposedly Apple fans. I remember Phil Schiller mentioning the "naysayers" when introducing the iPad and you know, there are as many naysayers in the Apple community as there are on the PC side. And that's too bad considering that Apple overcame so many obstacles to prove they can reinvent what others can't.
The bottom line is this: if Apple decides to introduce a 3D environment on iOS devices, they'll do it right! Some of us know Apple's history of spitting in the faces of the naysayers. And I for one, absolutely love it!
Posted by: MonkeyMo | January 13, 2012 at 04:43 AM