Earlier this month the U.S. Patent and Trademark Office officially granted Apple a patent that relates to a mixed reality system including a head-mounted display (HMD) and a base station which could be a Mac, iPhone/iPad or game console to play VR content and more. The headset is equipped with a wide range of cameras that are both interior and exterior to capture both the real world and users' expressions, eye movements hand gestures.  

Apple's granted patent, that bypassed the application phase under "Apple" as the assignee, covers methods and apparatus for providing mixed reality views to users via a mixed reality MR Headset (HMD) or smartglasses much further down the road.

The system described includes the MR headset connecting to a separate computing device via a wired or wireless connection.  

The HMD may include world-facing sensors that collect information about the user's environment and user-facing sensors that collect information about the user such as depth information, lighting information.

The user-facing sensors of the HMD collects different information such as expressions, eye movements and hand gestures. The information collected by the sensors may be transmitted to the base station via the connection.

The base station may include software and hardware configured to generate and render frames that include virtual content based at least in part on the sensor information received from the HMD via the connection and to compress and transmit the rendered frames to the HMD for display via the connection.

The base station, according to Apple, could be an iMac, MacBook, iPhone, iPad, game controller and/or game system. Whether a gaming system will include VR content from Xbox, a PlayStation or a future VR-Apple TV box isn't clarified.

Apple notes that in some embodiments, the motion information used by the encoder on the base station to encode a frame may be embedded in the data stream sent to the HMD along with the frame data.

Two primary constraints to be considered on the connection between the HMD and the base station are bandwidth and latency. A target is to provide a high resolution, wide field of view (FOV) virtual display to the user at a frame rate (e.g., 60-120 frames per second (FPS)) that provides the user with a high-quality mixed reality view.

Another target is to minimize latency between the time a video frame is captured by the HMD and the time a MR frame is displayed by the HMD.

In some embodiments, the world sensors may include one or more world mapping sensors (#221 e.g., infrared (IR) cameras with an IR illumination source, or Light Detection and Ranging (LIDAR) emitters and receivers/detectors) that, for example, capture depth or range information for objects and surfaces in the user's environment. The range information may, for example, be used in positioning virtual content to be composited into views of the real environment at correct depths.

In some embodiments, the world sensors may include one or more head pose sensors (#222 e.g., IR or RGB cameras) that may capture information about the position, orientation, and/or motion of the user and/or the user's head in the environment.

The information collected by head pose sensors may, for example, be used to augment information collected by an inertial-measurement unit (IMU #206) of the HMD (#200). The augmented position, orientation, and/or motion information may be used in determining how to render and display virtual views of the user's environment and virtual content within the views. For example, different views of the environment may be rendered based at least in part on the position or orientation of the user's head, whether the user is currently walking through the environment, and so on.

Apple's patent FIG. 1 below illustrates a mixed reality system #10 on a user with a headset (HMD) viewing virtual content; FIG. 2 illustrates world-facing and user-facing sensors of a head-mounted display (HMD) in a mixed reality system as illustrated in FIG. 1.

The number of cameras in the MR Headset is quite extensive which supports a report from The Information earlier this month that described an HMD prototype having 12 cameras. More specifically, The Information report stated that "A mixed-reality headset Apple is developing will be equipped with more than a dozen cameras for tracking hand movements and showing video of the real world to people wearing it, along with ultra-high-resolution 8K displays and advanced technology for tracking eye-tracking technology." The basic design of the prototype in that report is presented below.

Back in June 2020, Mark Gurman writing for Bloomberg reported on Apple's secretive Headset team and described two kinds of wearables, a VR Headset and Glasses. The former was to be a crazy powerful headset that connected to a Mac or high-end station that got vetoed by Jony Ive. But with Ive gone, one has to wonder if the higher end VR headset could be back in some form. This patent supports a headset taking advantage of higher-end processors in a "base station."

Apple's granted patent goes on to describe matters better suited for developers and engineers in segments titled as follows:  

• Bandwidth and Latency Constraints on the Connection
• Encoding Method
• Embedding Motion Information in the Data Stream to the HMD Over the Connection
• Example Mixed Reality System
• Layer-Based Rendering and Encoding
• Layer-Based Rendering, Encoding, Decoding, and Processing
• Variable Degrees of Compression
• Rendering Scene Camera Frames on the Base Station

For the finer details, review Apple's granted patent 10,914,957.