Microsoft's Glass Project for Live Events

Fans of live sporting and artistic events have enjoyed various types of information which is provided on different displays to supplement the live event. This supplemental information is provided both when the user is in attendance at the event and when the user views the event on broadcast media. One example of this is the augmentation of football games to display a first down marker superimposed on a playing field in a broadcast television event, and the broadcast of replays both on broadcast television and large displays at the venue of the event.

Up until now, Microsoft's inventions regarding head mounted devices have been focused on gaming, general virtual reality applications and an innovative one related to Life Streaming which could end up being an interesting one in the future.

This week's published Microsoft patent application indicates that they're looking to expand their headset project parameters a little wider into the consumer market. Microsoft's latest invention relates to advancing a head-mounted display system in the form of glasses focused on providing users with a new live-event experience.

Microsoft's invention relates to technology that will provide a user wearing a head-mounted display with supplemental information when viewing a live event. The information is provided about actions and objects occurring within an event and within the user's field of view. A user wearing a partially see-through, head-mounted display will be able to view a live event while simultaneously receiving information on objects, including people, within the user's field of view as noted in one of our graphics below.

In a further aspect, a computer implemented method providing supplemental information to a user with a head-mounted display viewing a live event includes receiving an indication that a user is attending a live event. The user's field of view through the head-mounted display is determined and objects within the field of view at the live event identified. The supplemental information is displayed in the head-mounted device and may change as the user's view changes or objects and actions within the live event change.

As noted in our second Microsoft patent figure, one of the cool features will be providing users with "instant replays" of important actions in a game, be it a goal in a hockey game, a homerun in a baseball game or an important or controversial call in any game. Logic tells us that over time, augmented reality based glasses will be able to tell you who's on stage at a concert and provide you with the lyrics of the songs being played. In fact, although we don't show that graphic in this report, this feature is presented in Microsoft's new patent with a note that if you're watching a live event like a live foreign-language Opera, you'll be able to receive an augmented translation of what's being sung on the lens of your glasses.

As noted in our graphic above, Microsoft is thinking of using a wired or wireless wrist band to work with their glasses. The wrist band would be where the computer and power would be contained, using the glasses as an intelligent accessory. This wrist band is also noted in Microsoft's "Life Streaming" patent graphic noted above.

Below is a patent figure illustrating augmented reality relating to live baseball game with instant replay capabilities.

Kopin Corporation, a Rising Star in Glass Projects

There's no doubt that Glass Projects from various companies will be emerging as one of the next battlefields in consumer electronics over the next five to ten years. In May of this year we pointed out that Google was advancing their Glass Project by experimenting with a connection to using consumer magic rings and tattoos. In June we presented Sony's take on a Glass Project relating to a smartphone which is the smartest idea to emerge on this kind of device thus far – and earlier this month we showed you that Samsung is experimenting with various brain computer interfaces that could utilize some form of next-gen glasses or a head-mounted display system which is likely too many years away to care about now. Yet all in all, it shows that the industry is in a race to deliver a hot next gen product in the form of a video headset.

At the end of the day, many of these high profile companies will be working with OEM's that specialize in this field to help their Glass Projects get off the ground. One of the leaders in this field is Kopin Corporation and earlier this month the US Patent Office published one of their latest patents in the area of future head-mounted display applications.

Kopin's patent FIG. 2 depicts a viewport into a 3-D virtual space that may be manipulated with voice, head motions, or hand gestures. In this figure we're seeing an engineer reviewing the schematics of a building and site.

In this scenario, the engineer is able to specify a position and magnification and/or zoom level to be applied to a particular application software window. The result is similar to using a magnifying glass to look at something seamlessly over a large area.

According to Kopin's invention, the user will also be able to issue commands to retain a piece of a large image that they wish to magnify and freeze a portion on the screen or request another level of magnification for a particular area of a schematic or map.

In yet another example, the user will be able to issue voice commands to manipulate the position of the various image objects in the 3-D virtual space. For example, a user may select an image object such as by moving their head, but then issue a voice command such as to "move object up" or "move object A behind object B". This causes the head tracker to then control the relative position of the selected image object(s) within the 3-D virtual space, rather than allowing the user to navigate among a given single object within the 3-D space.

Kopin notes that it will be understood that the headset wearer will also be able to access a virtual desktop that is in any form factor that could be represented in a 3-D virtual space.

A key market for Kopin's technology is the military. In fact their website states that their "CyberDisplay LCDs are the microdisplay of choice for advanced night vision goggles and thermal weapon sights used by the U.S. Army."

In their most recent patent, they point to technology that could work with new military applications that I'm sure could also apply to civilian SWAT units as well. The speciality head-mounted systems could be used in bomb squads to control robots and/or miniature unmanned aerial vehicle drones to survey areas of potential conflict. It's also noted elsewhere that the headsets could be modified for gaming.

The headsets from Kopin are noted to possibly work with any number of sensors including atmospheric sensors, health condition sensors, GPS, digital compass, pressure sensors, environmental sensors, energy sensors, acceleration, position, attitude, motion, velocity or optical sensors, and cameras with infrared capabilities. The headsets will also be equipped with WiFi and LTE wireless capabilities.

As a side note, it was interesting to read that Kopin describes one aspect of their next generation of video classes as including "an eye pod assembly" which is to include a microdisplay, one or more microphones and one or more speakers.

Where is Apple on this Emerging Trend?

At the end of the day Patently Apple is noticing that the number of new patent applications that are popping up at the US Patent Office regarding head-mounted display systems are on the rise from companies like Microsoft, Google, Sony and others. And, considering that Google's Project Glass is scheduled to make its debut sometime in the 2013-2014 timeline, it's going to push other tech leaders to start prototyping new headset systems of their own at an accelerated pace.

This class of device has caught the imagination of the world. At the beginning of November, Time awarded it the invention of the year. Over the years, Time Magazine has promoted Apple's latest new iDevices and in fact five years ago this month the iPhone was the invention of the year. And so Time's endorsement for Google's Project Glass is meaningful. You also know that a trend is hot when it's touted on the runways of a New York fashion show as it did this past September. Sergey Brin, Google's cofounder and the head of Google X was on stage to show off Project Glass at Diane von Furstenberg's fashion show.

The attraction to Project Glass is that it holds so much potential over time. It's trendy, it's futuristic, it could be used as a next generation smartphone, it could be the next craze for camera buffs. I mean why carry a camera when your trendy glasses could capture what you're actually seeing? Filming a concert will be a synch without having to hold up your smartphone. You could walk and film any situation at the same time naturally without any bother. It will augment live events, will double as an MP3 player and holds much more promising applications on the consumer side. Advanced head-mounted display systems are and will continue to play a role in the military and extend to law enforcement agencies in the next decade. It will find industrial applications as we noted in our report. Its potential is truly explosive.

So where is Apple on this trend line? Well, Apple began working on a head-mounted display system back in 2006 and their original work was published by the US Patent Office in 2008. That very same patent was granted to Apple in 2009.

A second patent came to light the year after and although its implementation looked rather awkward on paper, one has to remember that patent applications aren't design patents, and so the appearance of the video headset patent figures have no bearing to the final product. Apple patent filing showed a lot of potential even if it was a little unfocused.

No other patent application on this project has surfaced since 2010 and one of Apple's engineers whose specialty is wearable computers has since departed Apple and moved to Google. Did Apple prematurely give up on their video headset or is the project simply in limbo?

Without Steve Jobs' vision for a few killer apps to accompany their base of iDevice apps for a video headset, it's unknown whether or not Apple has what it takes to pull off such a device successfully. The Crazy Ones in Cupertino are extremely talented engineers but they were always guided in some way by Steve's prodding and talented input.

The fact that Tim Cook has implemented project "Blue Sky" is encouraging on one hand. On the flipside however, some think that its very implementation is stating that Apple is already running out of ideas while others think it's a new day of collaboration at Apple. The jury is still out on the latter. 

And don't underestimate the impact of Scott Forstall's departure in 2013. Forstall wasn't just a VP of Software at Apple – he was one of Apple's top inventors and likely one of the top thinkers at Apple reporting to Steve Jobs directly. So in the short term, a temporary void in creativity could impact Apple's ability to deliver all-new kinds of iDevices needed to keep Apple flying high. Everything Apple delivered this year, besides their new earphones, was simply a reinvention of their baseline of products. The iPad mini was great, but it's still just an offshoot of a standing product. It's evolutionary at best. That's great, make no mistake about it, but that still leaves a question mark as to what new device is on the horizon. Even an HDTV from Apple appears to be in limbo.

Prior to the iPhone, wireless iPod and iPad coming to market, there was a long seven year trail of patent applications forecasting such devices were on the drawing board. Steve Jobs even touted during the unveiling of the iPhone that it was protected by more than 200 patents. No such trail or trend yet exists for video glasses.

Apple was one of the first high tech companies to patent video glasses but it's going to be Google who wins by running with the idea instead of trying to be perfect out of the gate. Google's Android in my view, is inferior to iOS on appearance alone. But now Android is the number one mobile OS in the world. The point being that sometimes Apple over thinks projects and playing catch up to an idea that they had first will be a crying shame. Yes, Apple could release a video headset down the road and it will likely be better. But if the Android ecosystem has too long a lead, Apple might be limited to selling it to their existing fan base. Some will think that's good enough while others will see it as a sign of Apple's leadership being chipped away. It's already the sentiment on the street in many consumer circles that Apple is in decline.

Although I'm hoping against all odds that Apple has a secret skunkworks project in play behind the scenes, for now it appears that they'll be playing catch up to Google's Project Glass which is likely to spill over to similar products from Samsung and other Android developers rather quickly.

In our October report titled "Apple, Google & Others Preparing for the Coming Glass War," we quoted the forefather of wearable computing Steve Mann as stating that "Yes. There will be Apple Glass, and Google Glass, and RIM Glass. These companies are all working on glass. I think everyone is going to be making glass. I think we're also going to have a glass war instead of a smartphone war." Products like Glass will introduce simplistic elements of the brain-computer interface.

In the end, it's an open question: In the coming wave of highly advanced wearable computers, will Apple lead or follow this trend? Will a next-gen iPhone-centric video augmented headset surprisingly emerge prior to the launch of Project Glass to steal the limelight? I don't know, but I think there's a few million Macites around the world that are hoping for a Rocky Balboa surprise that could make the hair on the back of our necks rise in shear surprise of such a product.

But let's not kid ourselves, it's a long shot at best.