The Next OS Revolution Countdown Begins
Anyone who had taken the time to tune into the Steve Jobs interview with Walt Mossberg and Kara Swisher at this year's D8 Conference, likely walked away with that one special quote that stood out from all the rest. For me it occurred at the 7:05 mark when Mossberg asked Steve Jobs if he thought that the tablet would succeed the laptop. Jobs replied "The transformation of PC to new form factors like the tablet is going to make some people uneasy because the PC has taken us a long ways. The PC is brilliant…and we like to talk about the post-PC era, but it's uncomfortable." Yeeesss, more changes are in the wind and in fact, we're likely on the cusp of the next OS Revolution - and I'm talking about the OS X revolution here! This report will present you with some of the great innovations that are on the drawing board at Apple and elsewhere in the industry. Before Apple's WWDC gets under way next week, why not take a look at what could be coming our way in 2011 - because it's going to be wild!
The Power of the User Interface
In December 2009, Patently Apple posted a report titled "Apple Mages Working on Augmented Reality Magic." Our report covered a number of key developments that are in various stages of development. One of them involved a user interface built right into a washroom mirror at a Vancouver hotel that will eventually filter down to the average home user over the next five to ten years. While the initial UI may have been limited to a TV UI, the fact is that it will eventually take on a few other apps like email and voicemail. In the morning, you want to be able to hear the top news headlines, find out what your local weather will be like and perhaps catch up on any urgent emails that await you.
In that same report, we covered Apple's patent pending Head Mounted Display System, tabletop computers for home and work and a device called the SixthSense. In the SixthSense's presentation by MIT's Pranav Mistry, you were able to see a futuristic iPod-like device working with full hand gesturing that somewhat reflected the hand gesturing found in the movie called "The Minority Report."
In today's report we continue from that exact point to present you with the man and company behind the real Minority Report Style interface, John Underkoffler. The revolutionary interface that he developed at MIT is called the "g-speak - Spatial Operating Environment." He recently presented his work at a TED event and certain aspects of his presentation will be reviewed in context with this report.
Very early into Mr. Underkoffler's presentation he uses the Mac OS as the core of his initial point, as follows: The early Macintosh team in '82, '83 …'84 had to write an entire new operating system from the ground up. Now this is an interesting little message and it's a lesson that I think has since been forgotten or lost or something – and that is namely … that the OS is the interface. The interface is the OS. It's like the land and king in Arthur, they're inseparable, they're one. And to write a new operating system wasn't a capricious matter. It wasn't a matter of tuning up some graphics routine - there were no graphic routines. There were no mouse drivers."
While respecting the pioneering spirit of the original Macintosh development team, Mr. Underkoffler's point was blunt. After twenty-five years with one form of operating system, it was time for a next generation OS to emerge - One that was a magnitude higher than that of the Mac OS. He added that the dramatic rise in computer power has made it possible to now raise the bar on the OS and once again ignite the next computer revolution.
In one particular segment of Mr. Underkoffler's presentation he had his colleagues join him on stage to cooperatively work on a CAD program. His two colleagues are seen in the graphic below using next generation 3D work wands. That particular example struck a chord with me because of two recent Apple patents. The first one describes the very same style of 3D wand and the second patent describes an NFC capable iPhone or iPod touch interacting with with a CAD program in the very same manner as was presented by Underkoffler and team.
You could fast forward the video below to view this particular video segment beginning at the 10:15 minute-mark - though the entire presentation is definitely worth reviewing so that you could learn about some of the underlying principles behind the next computer revolution.
At the end of Mr. Underkoffler's presentation, Chris Anderson, curator of TED, asked the obvious question: Can this be for everyone or is this just for corporations and movie producers?" Underkoffler emphatically replied, "No it has to be for every human being. That's our goal entirely. We won't have succeeded unless we take that next big step. I mean it's been 25 years: Can there really only be one interface [referring to the Mac OS model UI]? There can't."
Anderson continued his questioning with "Does that mean that at your desk at home you're going to need projectors, cameras … I mean … how can it work? Underkoffler replied, "This stuff will be built into the bezel of every display; it will be built into architecture; the gloves go - in a matter of months or years. This is the inevitability of it."
Underkoffler is convinced that within 5 years, this next generation UI will be standard on PCs. That's obviously a very bold forecast. At the higher end of the market Underkoffler pointed out that the system is now being field tested in areas such as logistics and supply chain management, in the natural gas industry, resource extraction, financial services and other computationally demanding fields. Filtering this down to the consumer desktop will be quite the task. Though admittedly, Apple is working on something similar in principle that we'll touch on later in this report. For more information on g-speak and other related technologies, check out: Oblong Industries Inc.
Evolutionary Change: A Hybrid UI for the Desktop
While one could appreciate Mr. Underkoffler's fifteen years of dedication and passion for his g-speak project, I think that his revolutionary OS is a little too radical for his forecasted time frame. And yet he makes a strong case for taking the UI to the next level. Which brings us back to our opening reference of Steve Jobs being uncomfortable about pushing us into the Post-PC era too quickly. Although I'm paraphrasing, that's really what his statement boiled down to.
While people love to talk revolution, they secretly abhor rapid change in their daily lives. I've seen that in my life when I pushed radical ideas to those in manufacturing or in sales. I see how people react to some radical Apple patents. When Apple patents emerged in Q1 2006 describing a new wave of tablets, a very large contingent within the Mac community all but screamed heresy! Apple failed at the Newton and they'll never try that again, they railed. Others proclaimed that a tablet was a stupid idea that would never fly! A year later Steve Jobs presented the iPhone. Radical? Most certainly. But the idea of change scares many in the community and Steve Jobs knows that all too well. Apple knew that their patents were being covered in the community from reporters such as I and that awareness leads to generating debate and discussions and it's a way and/or means of preparing a community for the changes that lie ahead - and it did.
Some projects have to be evolutionary so that fear could be controlled or harnessed. That kind of balancing act is an art form in business and few possess that brilliance. Once Apple's iPhone was a smash hit and people were comfortable with a revolutionary new touch UI, they knew that the iPad would be an easier sell. Were they right? Yeeesss, of course.
You also have to appreciate how Apple is evolving the iPhone OS at the moment so that the masses could get used to new features and more importantly the next wave of advanced hand gestures in one, three, five new gesture-lots at a time. You only have to look at some of Apple's hand gesturing patents to fully appreciate that there are many new gestures on the way in addition to expanding gesturing further into virtual keyboards and even trackpads.
Today, Apple has intelligently integrated simple gesturing into their desktop via their Magic Mouse and have brought gesturing to their MacBook line-up via a unique Multi-Touch trackpad. Yet we also know that Apple is planning on some form of iMac Touch because we're read several of their patents describing various approaches for achieving this.
An iMac Touch would serve many well. In the home, it would be a great addition to a family room or kitchen where accessing media like iTunes, iPhoto or other apps would be quick and easy with a simple tap of an icon. For a small business, like a travel agent in a mall, lining up a few cool iMac Touch units in their store window to lure customers in to check out their travel videos with a simple tap on a destination icon or photo would be great in drumming up business. But would adding a touch screen to an everyday work iMac really be the best approach for advancing gesturing on the desktop? Perhaps not.
So what could be a better solution? Well, let's take a look at an alternative solution for addressing a hybrid UI that would further incorporate multi-touch into OS X. Keep in mind that the following is simply an overview of a theoretical hybrid UI. It's not from Apple and it's definitely not Gospel. Simply consider it a visual exercise to stimulate conversation between your friends or community. What works and what doesn't? What aspects of this could Apple integrate into such a Hybrid UI? Just have some fun with it.
The Continuum Project
The following is from the Continuum Project video: "The predominant mode of spatial human/Computer interaction for the past quarter century has proven itself powerful and flexible. But is the mouse to be indelibly the sole conduit for desktop interaction? Despite the Mouse's pedigree it's restriction to a single point of interaction creatively limits the bandwidth of information that it could convey."
The video goes on to state that there's "a rough consensus in the industry that the future lies in the domain of multitouch technology." The twists that are presented in the video are quite the eye opener to where touch could go on the desktop. The video logically traverses through a series of sections explaining the current problems with today's desktop environment in context with hardware and software and then finally it goes on to the theoretical next generation desktop that will push us through the next few decades.
The project name sounds like continuum, yet the official logo is presented as "Con10uum." So we'll refer to it as such going forward. Con10uum organizes files in a new linear manner. Local and global edges are added to a new touch pad associated with the keyboard as we've note above in our graphic. That design acts a means of controlling and managing applications. Theoretically it could replace the current application dock or perhaps simply provide an alternative methodology for keyboard lovers.
A Different UI is Required to Properly Enhance the Power of Multi-Touch
The Con10uum Video
The Con10uum video presents a few novel UI conventions worth reviewing. To accommodate the next generation of multi-touch interfaces that will make us more productive, there has to be a reinvention of some aspects of the UI of today. There are definitely a few great ideas presented in this video that Apple could very well have on tap in some shape or form – which makes this an important video to review when you have the time. Although the concepts are theoretical, I think that you'll agree that they're also quite plausible. The screenshots noted above are presented in the following video.
The idea of creating a next generation keyboard that would allow for greater Multi-Touch freedom is rather a fascinating idea within the spectrum of yet other conventions that could make their way to the desktop in the future, be they voice commands or full hand motions as were presented earlier by Oblong Industries. The interesting fact in all of this, is that Apple has a patent supporting a wider touchpad that could definitely be incorporated into both the MacBook and a future upgraded Desktop keyboard so that there's a common methodology or continuity for users to follow regardless of device.
Apple's next generation touch-pad isn't just about it being wider, it's about the fact that Apple is going to incorporate some form of coordination between the trackpad and the iSight camera as noted in the patent figures. This could be for free-form full-hand gesturing as Oblong Industries presented or something not yet defined.
While we're on the topic of advanced desktop keyboards, we could also hope to see Apple adding iPhone-like functionality like the new ASUS keyboard. I mean, who could seriously look at that keyboard and not see where ASUS got their inspiration from. But they got to it market faster than Apple, so kudos to them. Yet let's hope that Apple catches up on that kind of feature. Alternatively, Apple could integrate an iPod touch or iPhone docking station into the right side of the keyboard so as to provide us access to our iApps and more on the desktop.
Perspective and Revolutionary Shifts
When it comes to operating system developments, it seems that all we hear about in the press of late is iPhone OS 4. Yet it stands to reason why Apple is pushing iPhone OS 4 so hard; being that Apple's CEO declared the company the number one mobile devices company in the world during his January keynote. If you're the leader of a particular sector due primarily to a given OS, how could you do otherwise? It's Apple's iPhone OS 4 that will power the next generation iPhone, iPod touch and iPad - and that's only the beginning of the mobile device revolution.
In the big scheme of things, the industry's growth is going to come from a plethora of smart devices be they mobile or not and it's the iPhone OS 4 that will be Apple's leading star. Think about the OS that will run your future internet TV, or your future in-vehicle multimedia system or a cool head-mounted display in the form of sunglasses, or your e-wallet or a possible next generation gaming console. Apple's iPhone OS is Apple's future.
Yet with all of these revolutionary devices coming to market, will it mean that Apple leaves OS X to die on the vine? Many think so, but I'm definitely not in that camp. On one side, Apple could evolve the OS X UI to include more advanced gesturing. On the other side, Apple could definitely present us with a rather radical 3D version of OS X based on advanced Head Tracking Technology that will likely arrive once Apple introduces next generation 8 and 16 core systems. 3D technology is going to take a lot of desktop power and we've discussed this in terms of the 3D Internet, a likely 3D version of iPhoto or Aperture and a Virtual Apple Store. Apple's OS X die on the vine? Ha - not a chance!
We could also look to the wild hand gesturing system developed by Oblong to know that such a system could challenge Apple's OS X. Yes, it could. But it's not like Apple is sitting back as the world goes by. Na-na-na, Apple has shown us through a recent patent that they're already working on tracking technology for the display bezel as we highlighted Mr. Underkoffler stating. If Apple's tracking technology could track ambient light, don't you think it could track a solid hand gesture? Of course it could. Every patent tells you that the concepts protected could go far beyond those examples presented. And if you want to talk radical, how about Apple's recent desk-free computer. So be rest assured that the Crazy Ones in Cupertino are on top and ahead of industry trends and will get us there in style.
At the End of the Day
At the end of the day, with all of the advances that we've discussed here today, the question simply boils down to one: when will we see some of these advances come to OS X?
I'm of the opinion that we'll finally witness some rather dramatic shifts coming to OS X sometime in the second half of 2011. It appears from watching recent presentations from both AMD and Intel that we're going to experience a massive leap in power and capabilities from both their respective Fusion and Sandy Bridge platforms in 2011. And the one technology that rang out in every presentation was none other than OpenCL from Apple: The very same OpenCL that is buried within the heart of OS X - today. The technology that will harness the power of these incredible CPU-GPU rockets for next generation operating systems, browsers and applications.
It's coming, get ready, the next OS revolution countdown begins on Monday.
Our Report is also Being Covered By: MacSurfer, Apple Investor News, 9 to 5 Mac, CALMUG Calabria Italy and more.
Update June 7, 2010: Related Material: Apple Files for "Magic Trackpad" Trademark. Also see Engadget for Magic Trackpad concept photos.
Apple excels at transitions. 68K to RISC, RISC to Intel, OS 9 to OS X, and now mouse+keyboard to multi-touch + something new . The key to Apple's success in transitions lies in 1. developing a clear plan and committing to it, 2. helping developers program apps to the new technology, and 3. easing users into the new technology.
I think Apple has done a brilliant job designing and committing to multi-touch on iOS over the years. And since the underlying Cocoa frameworks are very similar, if not identical, in both iOS and Mac OS X, it is relatively easy for Mac developers to transition to iOS programming. And finally, by introducing a simplified multi-touch before the full-blown version (which we likely haven't even seen yet), Apple has eased users into the concept.
All of which helped iPad launch to a ready and willing audience. If Apple had simply dumped the iPad as it is now onto the unsuspecting public in January 2007, it might have sold 2 million units in 2 years instead of 2 months. The concept would have been such a drastic leap forward that only techies could grasp it. Instead, Apple introduced a simplified iPhone multi-touch UI, which immediately spawned copycat multi-touch OSes and devices. Which both validated the concept and conditioned the world for further multi-touch innovation in iPad and likely in future products.
Just from a marketing label standpoint, maybe Apple should call their next desktop OS "11" or "XI" instead of the expected "Mac OS X 10.7". If it really does have some all-new technology, or even if it just has some multi-touch technology from iOS, maybe it's time for a new name.
Posted by: Boso Zoku | June 11, 2010 at 11:56 PM
The PC will not disappear. Sorry, but I'm adding workstations, not removing them. When I get an iPad it will be to compliment my work and locale, not to replace it. The same goes for the iPhone 4 when it arrives, I will let Apple bang out some of the bugs and get one when iPhone OS 4.1 arrives. It's a tool not a lifestyle.
Posted by: Marc J. Driftmeyer | June 05, 2010 at 11:24 PM
Maybe just one post...
What a lot of people understand is that iPhone OS X is a stripped-down Mac OS X that still retains its UNIX underpinnings... You can Install and run an Apache Web server on a [JailBroken] iPhone. Yes, there are a lot of [unnecessary for mobile] things removed from iPhone OS to keep it lean and mean (mouse drivers, windows, etc). Then iPhone OS X has a few things added-- Touch Interface, Location Services, Compass, Cell radio, etc.
What many don't realize is that Mac OS X iPhone OS X is a two-way street. Many of the things that have been implemented (or re-implemented) in iPhone OS X are migrating back into Mac OS X.
At some point in time, the developer may be able to deliver an Universal app that runs on the Mac desktop, the iPhone/Touch/iPad and the AppleTV (and such).
Posted by: Dick Applebaum | June 05, 2010 at 07:21 PM
Way to go, Jack!
This article is so loaded that I don't know where to start...
...so I'll just post links to it!
Posted by: Dick Applebaum | June 05, 2010 at 06:17 PM
As a person with disabilities, I have long hoped for Gui 2.0. The original Mac empowered me for many years then Apple wandered away from pushing the boundaries. Now, hopefully, Apple can show once again they are the crazy ones. I am waiting for ipad 2.0 to solve some problems before I can do stuff I dreamed about for 25 years to do.
Posted by: Meitnik | June 05, 2010 at 04:54 PM
G-Speak reminds me of a theme park ride. While it may well have application in the professional 3D application area and gaming, it appears too close to Microsoft's Surface for comfort. I do agree that we are closing in on a fundamental shift which will debut as a separate OS from X on the desktop. OS X may well adopt multi-touch and other advances until it's end of life. I can see it's retirement coming and wonder what Apple will have as an alternate to it's BSD Unix foundation. That, of course, is separate from UI developments. AI will no doubt shape both.
Posted by: RBoylin | June 05, 2010 at 11:25 AM
The idea of the touch expanded keyboard in context with Apple's patent shows that they're toying with this idea. Good catch on associating the ambient light patent with gspeak's bezel admission. Very cool. likewise on the wand. Great Saturday morning read.
Posted by: James | June 05, 2010 at 10:57 AM
@Paul. Huh? I'll always want a real keyboard on my desk, thank you very much, and either creating a simulated iPod touch beside the keyboard or adding a dock (even better) would be sweet. Do away with the keyboard on the desktop, get real Paul. To each his own I suppose.
Posted by: Ducky | June 05, 2010 at 10:49 AM
Why would Apple be impressed with or try to imitate the ASUS Keyboard? It's not portable, and merely replaces the mouse with an iPod. It implements perfectly the derivative "innovational" practices of Asustek. Apple's idea is to get rid of the keyboard and mouse interface, not institutionalize it. The agenda is to improve the virtual keyboard or replace it with spoken or icon-based-gestural inputs.
Posted by: Paul Johnson | June 05, 2010 at 10:41 AM