Over the last year, a number of very interesting Apple patents have come to light concerning various technologies that at the time of their publication were simply seen as islands of information unto themselves. Yet in hindsight, I can now see that yet another major Apple trend has emerged in 2009, that of Augmented Reality. I didn't recognize the extent of Apple's involvement in this area of technology until I had read a recent report filed by Nikkei Electronics Asia on this very subject. Once I saw the examples that were being presented in that report, the related Apple patents were easily recognizable. This report takes a peek at the magic that the Cupertino Mages are now conjuring up for us in future products and wonder if we're not actually looking at the beginning phases of some kind of new sixth sense.
A Revolution on our Door Step
The introduction to Nikkei Electronics Asia report begins with an enthusiastic bang by stating that "Augmented reality (AR) technology is coming into our homes, our cities and elsewhere in the near future. Dramatic advances in computer technology are making possible a world that once would only have been found in the realm of science fiction... The revolution is impacting everything from mobile phones to digital home electronics, forcing rapid evolution."
Technically, "Augmented reality (AR) is a term for a live-direct or indirect view of a physical real-world environment whose elements are merged with (or augmented by) virtual computer-generated imagery - creating a mixed reality," according to Wikipedia. Some of us are aware of augmented reality through something as simple as the NFL's yellow line – though it's anything but simple technically behind the scenes. Others are experiencing simplistic AR apps like that of Wikitude or Layar (Layar video) for Apple's new iPhone 3GS. And yet others are simply able to take a shave while watching the big game highlights on their bathroom mirror. Today you could actually experience that at the Shangri la Hotel in Vancouver Canada, as noted in the report's opening graphic. The mirrors are designed by Electric Mirrors out of Seattle. Within the next five years or so, we'll begin to see these mirror based televisions rolling out into new luxury homes and then eventually filtering down to the general public – if you care that is. That said, augmented reality applications are going to ramp up and stream out over the next decade with some very cool and technically important applications. Technology that may have once begun with military applications such as night vision goggles and fighter pilot targeting displays is now entering new important fields like robotic surgery. Yet the good news for most of us is that value-added augmented reality will work itself down into more common every day products like Apple's iPhone in the not too distant future. Professor Hirokazu Kato of the Nara Institute of Science and Technology (NAIST) of Japan, known for the ARToolkit C-language library for AR free of charge, pointed out that "Even the iPhone boasts the processing power of a workstation of the 1990s." (About ARToolkit, Download available for OSX, Linux and Windows XP). Yes, exactly, and with increasing power, Apple will be able to raise the technological bar in future iterations of their higher end media players. Just how high will they raise that bar? Well, we'll first take a look at a very cool MIT project in development. The presentation will definitely provide you with the necessary visuals required to understand some of the concepts that we could easily envision coming to Apple's media players. The SixthSense Pranav Mistry is the inventor of SixthSense, a wearable device that enables new interactions between the real world and the world of data. Mistry is a PhD student in the Fluid Interfaces Group at MIT's Media Lab. Below you will find both a collage of photos and a TED-India video presentation about the Thrilling Potential of SixthSense Technology. The SixthSense literally gives new meaning to the concept of a handphone as you could clearly see in the top right corner photo below. The SixthSense device, which can be seen in the top left corner photo, consists of a combination camera, projector and mirror. The hand gesture is recognized by the device and projects a phone pad onto the user's hand. The accompanying video will show you how easy it will be to make a call in the future when in situations where pulling out a physical phone may be inconvenient – like in a crowded bus, subway, classroom (shhh!) or car seat etc.
The middle and bottom left side photos are illustrating the user simply using a frame-gesture to take a photo with the SixthSense device. The middle and bottom right side photos represent the actual photos that were taken without having to position a physical camera.
As far as a camera goes, I think that the next generation concepts of SixthSense will likely include a Head Mounted Display that will be vastly superior. Once finalized, such a camera would naturally be integrated into the glasses or Head Mounted Display. This will greatly assist the user's frame-gesturing accuracy and simply be more intuitive than the current method employing a lanyard. But for now, however, one has to applaud the great pioneering work being done on the SixthSense by Pranav Mistry and team.
The SixthSense video provided for you below is really worth your time to view. There are some powerful concepts that you simply have to see in action, no matter how primitive, in order to understand how Apple could advance their future media players in context with augmented reality applications.
Science fiction is sometimes the inspiration behind motivating entrepreneurs to invent great things that the public are already comfortable with. For example, the race is on for a miniature video iPhone that resembles that of a Dick Tracy watch. Pranav Mistry's use of finger sensors in his presentation reminded me a little of the movie the Minority Report – where we see Tom Cruise moving documents around on this massive screen before him to piece together the facts of a case he's to solve. In contrast, the sensors used in concert with the SixthSense are real - however primitive they may seem.
Some of the concepts in the video from TED-India are already reality on Apple's touch based media players. The pinch gesture and a multitude of gestures are in place and are tried and true in the market today. Many of Apple's media players incorporate a camera and within a year, the top players will have a video camera – a vital component in the creation of a SixthSense-type of device.
The key ingredient behind the magic of a SixthSense device appears to be that of a unique video camera-pico projector combination. A mid 2009 Apple patent revealed that Apple has been working on this for more than two years. Patently Apple's report presents you with two videos about a pico projector and its simplistic interactions with the user. It could be used to project an iTune video or home movie unto various surfaces, be it a ceiling or even a friend's shirt, like you saw in the SixthSense video. In more complex implementations, we were able to see how Pranav Mistry used the device's pico projector to interact with paper in a mind-boggling way. It demonstrated the truer potential of a pico projector in the workplace or in a university campus application. Mistry simply pinched some text from a library book and then dropped it right unto a blank piece of paper. The text that Mistry pinched was recorded by the pico-projector and then projected to the blank paper. The application that was associated with the pico-projector created a form that included a virtual print button. Once Mistry gestured to touch that virtual button, the SixSense device wirelessly sent the text to the nearest printer available. That was simply stunning to behold. Of course I'm unable to discern what is marketing-magic from that which is real. Though I wouldn't be the one to bet against MIT. With a pico projector added to one of Apple's higher-end media players, the reality of such an application isn't that farfetched in the least. In fact I'd call this a potential killer app just waiting to happen.
So technically, Apple has the foundational technology in place today for a SixthSense-type of iPhone Application, with both the pico projector on the way and a Head Mounted Display system (HMD) firmly in development. Apple also has the widgets available that could be used with the displaying of a clock with a simple forearm-gesture as was shown in the video. It's Apple that has over 200 patents supporting their multi-touch technology. And with the iPod nano offering a built-in video camera this Christmas, it's Apple that in my view has the real-world momentum on its side.
Yet there's no doubt that it's going to be a race as to who gets to market first with the best SixthSense-type of device and applications. And if I were a betting man, I'd be putting Apple on my short list of potential winners – that's for sure.
The SixthSense presentation was the very best on the subject on Augmented Reality that I've seen to date in terms of providing us with a grand overview of the technology's potential. Pranav Mistry summed up his presentation this way: "So, as a last thought, I think that integrating information to everyday objects will not only help us to get rid of the digital divide, the gap between these two worlds, but will also help us, in some way, to stay human, to be more connected to our physical world. And it will help us, actually, not be machines sitting in front of other machines." Yes Pranav, how true.
Update: One of the best Head Mounted Displays that I've seen to date is shown below. Check out the lens area on your left and you'll faintly see a numeric pad on the inside of the glass. There's no doubt that many companies are now burning the midnight oil to find that right blend of technology and practically needed to make such a product viable. Yet one of Apple's strengths and advantages over the competition is their ability to design great integrated apps into a new device. Apple's iPhone magic rests with their great blend of multi-touch technologies and applications. A Head Mounted Display system will no doubt be powered by current iPhone applications but count on Apple to deliver just the right new apps to steal the show. In fact, count on it.
Other Potential AR Applications
I began this report by first noting a new report from Nikkei Electronics Asia and I'll finish my report with a few of their examples to once again link Apple Inc. to this coming AR revolution. Beyond covering the Head Mounted Display or HMD that we basically covered with the MIT presentation, there are several other examples of coming AR applications worth noting.
The photo collage above consists of a large marketing photo from Microsoft's Surface program along with two smaller photos from the Nikkei Electronics Asia report that point to Tabletop computers for the home market. Microsoft's Surface website provides us with a number of great videos to review so as to help us envision the coming reality of Tabletop computers for work and at home. Currently, Sheraton Hotels are offering their guests access to Microsoft's Surface tables to play with. This will definitely be a huge trend in the coming years. Although Apple has been fairly silent on this coming front, it must be noted that Apple has just been granted a virtual keyboard patent that happens to hint at both full desktop and wall surface computers. So is Apple working on surface computing? Yes, of course. Changing Home Electronics The Nikkei Electronics Asia report stated that "AR technologies … will cause a dramatic change in home electronics. Tonchidot's Iguchi, [chief executive officer (CEO) of Tonchidot Corp of Japan] for example, predicts that some types of home appliances will become extinct, replaced by AR. Home electronics interfaces like buttons will be replaced by the AR interface, and storage functions such as hard disk drive (HDD) recorders will be replaced by cloud computing. "Extinct" is a bit of an exaggeration, but it is quite possible that the cases, at least, will vanish." That's interesting being that a recent Apple patent covered Apple taking MobileMe to the Next Level. In that patent we were presented with a graphic figure connecting MobileMe to a home HDTV via Apple TV. That apparently, could very easily replace a DVR with a virtual solution, as noted by Iguchi. The Nikkei Electronics Asia report went on to state that "when AR first began to be applied in home electronics, the idea was that it would display additional information and take the place of the user's manual. As AR became more useful, though, the importance of real-world information based on weighted information dropped: the role of the hardware as the product interface shrank. In other words, the AR interface can take the place of not only the TV's buttons, but the TV display functions as well. "Interfaces that were once hardware - are being virtualized in AR," says Keio University's Inami." If we think back to the opening graphic of this report, we see that the electronic mirror can now morph into a television, that could likely morph into a painting and this is one of the ways that AR is going to invade our homes through our next generation High Def Televisions: Blending reality in with that which is virtual to provide us with a new AR experience. Apple's Virtual Store experimentation might be ready to launch as a product or service when we begin our journey into the world of the 3D-Internet that Intel and others are working on today. We'll all need to be avatars in that world and Apple has been working on that for years. But Apple may in fact decide to add more AR than VR so that your sales experience in Apple's Virtual Stores may blend in with a touch of reality in ways that we can't even imagine at the moment. The Bottom Line: The AR Journey I don't know about you, but at the end of the day, I certainly wasn't aware of how far along Apple's engineering teams really were in developing intertwining AR applications – until very recently. It's nice to know that there's actual proof that Apple is hard at work in developing the next wave of this digital revolution. The most promising development may in fact be a sixth-sense type of device in-the-making at Apple. Of course we'll simply know it as the next great iPhone – but nonetheless exciting. One of my favorite Apple patents has been the Apple Mounted Head Display - and who knew this was a key component in the AR world to come? In November I reported on the trend known as Millimeter-Wave Communications and now in December we can all clearly see that Apple is in full control of their role in the coming AR revolution. By following the publication of Apple's patents weekly and intently, we're able to see many interesting concepts and trends taking shape before they come to market – and Augmented Reality is simply the latest discovery. In many instances, Apple's Virtual Stores and virtual worlds in general, will require us to reinvent ourselves as a unique Avatar. We'll sit in front of our computers and allow Apple's iSight camera to build an Avatar profile for us - as we direct it to, of course. Then again, the iSight Camera just might have other plans for us as you'll see in this final video titled "Coke Zero Avatar AR."
Welcome to the World of Avatars. And to you that will be seeing the movie Avatar this weekend and over the holidays – Enjoy!
