It's being reported this afternoon that Apple Inc.’s long-anticipated mixed-reality headset is an ambitious attempt to create a 3D version of the iPhone’s operating system, with eye- and hand-tracking systems that could set the technology apart from rival products.

The roughly $3,000 device, due later this year under the likely name of Reality Pro, will take a novel approach to virtual meetings and immersive video, aiming to shake up a VR industry currently dominated by Meta Platforms Inc. It’s a high-stakes gambit for Apple, which is expanding into its first major new product category since releasing a smartwatch in 2015, and the company needs to wow consumers.

Apple is pushing into an uncertain market with a premium-priced product. The company’s 1,000-person-plus Technology Development Group has spent more than seven years on the project, and Apple is counting on it to become a new revenue source — especially with sales growth poised to stall this year.

Apple’s goal is to bring something new to the table. Eye and hand-tracking capabilities (covered in patents 01 and 02) will be a major selling point for the device, according to people familiar with the product. Its core features will include advanced FaceTime-based videoconferencing and meeting rooms.

The headset also will be able to show immersive video content, serve as an external display for a connected Mac, and replicate many functions of iPhones and iPads.

Here’s how it will work: The headset will have several external cameras that can analyze a user’s hands, as well as sensors within the gadget’s housing to read eyes. That allows the wearer to control the device by looking at an on-screen item — whether it’s a button, app icon or list entry — to select it." "Reading eyes" is covered by Apple in patents as technically being eye-tracking or eye-gazing. Here are a few patents covering their technology: 01, 02, 03, 04 and 05.

Mark Gurman further notes that "Users will then pinch their thumb and index finger together to activate the task — without the need to hold anything. The approach differs from other headsets, which typically rely on a hand controller."

Pinching a thumb and index finger together by itself would likely do nothing. Pinching a thumb and index finger together with Apple's patented accessory rings, as shown below, would make more sense in activating a user interface. Though alternatively, "in-air" gestures may be used and Apple has many patents on that technology here. 

In addition, Gurman notes that "Like Meta’s latest headset, Apple’s device will use both virtual and augmented reality. With VR, users see images and content within the goggles. AR, on the other hand, overlays digital content on top of real-world views.

The headset will have two ultra-high-resolution displays — developed with Sony Group Corp. — to handle the VR and a collection of external cameras to enable an AR “pass-through mode.” That means users will see the real world through the cameras positioned on the headset. Apple will offer users with prescription glasses custom lenses that sit within the enclosure itself.

The device will have a so-called Digital Crown — like the Apple Watch — that lets users switch between VR and AR. When in VR, the wearer is fully immersed." Apple's patent relating the Digital Crown with a Headset could be reviewed here.

Immersive video watching will be a core feature of the new device. Apple has discussed developing VR content for the platform with about half a dozen media partners, including Walt Disney Co. and Dolby Laboratories Inc. And the tech giant is working to update its own Apple TV+ material to work with the headset. As part of the push, Apple bought streaming company NextVR in 2020 (01 and 02), aiming to create sports content in VR.

Apple is planning for the headset to have a dedicated video-watching feature that can make viewers feel like they’re seeing a movie on a giant screen in another environment, such as a desert or outer space," according to Gurman.

It should be noted that a 2018 patent described a future headset being able to view movies, sporting events and concerts – and even be able to choose seating in a venue.   

Gurman adds that "The device will also have productivity features, including the ability to serve as an external monitor for a Mac. With that feature, users will be able to see their Mac’s display in virtual reality but still control the computer with their trackpad or mouse and physical keyboard." An Apple patent also shows us that a future headset could work with a virtual keyboard and be able to view 3D documents. The image below is from that patent.

Gurman further adds that "The headset’s operating system, internally called xrOS, will have many of the same features as an iPhone and iPad but in a 3D environment. That includes the Safari web browser, photos, mail, messages and the calendar app. And it will also have apps for the company’s services, such as the App Store to install third-party software, Apple TV+, music and podcasts. The company is working on health-tracking functions as well.

The experience should feel familiar to Apple users. When they put the headset on, the main interface will be nearly identical to that of the iPhone and iPad, featuring a home screen with a grid of icons that can be reorganized. Users will be able to pin widgets, such as the weather, calendar appointments, email and stock-market performance, among their app icons.

When users need to input text, they can use the Siri voice assistant or rely on an iPhone, Mac or iPad keyboard

Gaming is expected to be a popular offering from third-party developers, and Apple has created its own underlying engine to power VR games.

The headset can last about two hours per battery pack, in line with rival products. The relatively brief battery life could create a problem for those wishing to watch multiple movies or play games for hours at a time, they may need to buy multiple batteries and frequently swap them out.

To show off the new headset, Apple is creating a 'store within a store' concept — an area within its retail outlets dedicated to demonstrating the product.

The excitement for Apple's headset is building and for more on this, read the full Bloomberg report.

For more patents about Apple Mixed Reality Headset, view our archives here.