Apple Advances a future Mixed Reality Headset that could tap into the Power of a Base Station like an MX Desktop
Last week the US Patent & Trademark Office published a continuation patent from Apple that relates to a mixed reality system including a head-mounted display (HMD) and a base station which could be a Mac, iPhone/iPad or game console to play VR content and more. The headset is equipped with a wide range of cameras that are both interior and exterior to capture both the real world and users' expressions, eye movements hand gestures.
The HMD may include world-facing sensors that collect information about the user's environment and user-facing sensors that collect information about the user such as depth information, lighting information.
In some embodiments, the world sensors may include one or more world mapping sensors (#221 e.g., infrared (IR) cameras with an IR illumination source, or Light Detection and Ranging (LIDAR) emitters and receivers/detectors) that, for example, capture depth or range information for objects and surfaces in the user's environment. The range information may, for example, be used in positioning virtual content to be composited into views of the real environment at correct depths.
In some embodiments, the world sensors may include one or more head pose sensors (#222 e.g., IR or RGB cameras) that may capture information about the position, orientation, and/or motion of the user and/or the user's head in the environment.
The user-facing sensors of the HMD collects different information such as expressions, eye movements and hand gestures. The information collected by the sensors may be transmitted to the base station via the connection.
The base station may include software and hardware configured to generate and render frames that include virtual content based at least in part on the sensor information received from the HMD via the connection and to compress and transmit the rendered frames to the HMD for display via the connection.
The base station, according to Apple, could be an iMac, MacBook, iPhone, iPad, game controller and/or game system. Whether a gaming system will include VR content from Xbox, a PlayStation or a future VR-Apple TV box isn't clarified.
Apple's patent FIG. 1 below illustrates a mixed reality system #10 on a user with a headset (HMD) viewing virtual content; FIG. 2 illustrates world-facing and user-facing sensors of a head-mounted display (HMD) in a mixed reality system as illustrated in FIG. 1.
The number of cameras in the MR Headset is quite extensive which supports a 2021 report from The Information that described an HMD prototype having 12 cameras. More specifically, The Information report stated that "A mixed-reality headset Apple is developing will be equipped with more than a dozen cameras for tracking hand movements and showing video of the real world to people wearing it, along with ultra-high-resolution 8K displays and advanced technology for tracking eye-tracking technology." The basic design of the prototype in that report is presented below.
Apple's continuation patent goes on to describe matters better suited for developers and engineers in segments titled as follows:
- Bandwidth and Latency Constraints on the Connection
- Encoding Method
- Embedding Motion Information in the Data Stream to the HMD Over the Connection
- Example Mixed Reality System
- Layer-Based Rendering and Encoding
- Layer-Based Rendering, Encoding, Decoding, and Processing
- Variable Degrees of Compression
- Rendering Scene Camera Frames on the Base Station
The first inventor listed is a 22-year Apple veteran Geoff Stahl is Senior Director of Software, Technology Development Group which goes to illustrate the importance of this patent. The second inventor listed is Avi Bar-Zeev who left Apple in 2019. Bar-Zeev had been working in the AR/VR space for close to three decades. Back in the ’90s, he was part of a team at Disney that worked on some early VR experiences for the company’s theme parks, including "Aladdin’s Magic Carpet" VR ride.
While Apple was granted their first patent in 2020, the work had been started back in 2017. With Bar-Zeev gone, Apple decided to completely kill the original 21 patent claims and re-write them. Apple has added 20 new claims for this invention. Being a continuation patent, the new claims are numbered 22-41 with 1-21 cancelled. The numbers must "continue" and not replace the original numerically in order to keep their granted patent status. It's a technicality.
Apple's new patent claims are divided into 3 distinct segments of (1) HMD; (20) a Method; and (3) a System with each segment represented by a leading claim as follows:
Patent Claim #1: "A head-mounted display (HMD), comprising: one or more processors configured to: receive encoded frames from a device over a connection and decode the encoded frames; monitor the receiving of the encoded frames from the device over the connection and the decoding of the encoded frames to detect missing or incomplete frames; and upon detecting that an encoded frame is missing or incomplete, synthesize a frame based on a previously received encoded frame to replace the missing or incomplete frame; wherein, to synthesize the frame, the one or more processors of the HMD are configured to modify the previously received frame according to motion vectors received from the device with the previously received encoded frame."
Patent Claim #32: "A method, comprising: receiving, by a head-mounted display (HMD), encoded frames from a device over a connection; decoding, by one or more processors of the HMD, the encoded frames; monitoring, by the HMD, the receiving of the encoded frames from the device over the connection and the decoding of the encoded frames to detect missing or incomplete frames; and upon detecting that an encoded frame is missing or incomplete, synthesizing, by the one or more processors of the HMD, a frame based on a previously received encoded frame to replace the missing or incomplete frame; wherein synthesizing the frame comprises modifying the previously received frame according to motion vectors received from the device with the previously received encoded frame."
Patent Claim #40: "A system, comprising: a head-mounted display (HMD) comprising: one or more cameras configured to capture video of a user's environment; and one or more sensors configured to capture data about the user and the user's environment; an external device; wherein the HMD is configured to transmit the video captured by the cameras and the sensor data to the external device over a connection; and wherein the external device is configured to: render frames that include virtual content based at least in part on the video and the sensor data received from the HMD; encode the rendered frames according to a video encoding protocol; and transmit the encoded frames to the HMD over the connection; wherein the HMD is configured to: receive and decode the encoded frames; monitor the receiving and the decoding of the encoded frames to detect missing or incomplete frames; and upon detecting that an encoded frame is missing or incomplete, synthesize a frame based on a previously received encoded frame to replace the missing or incomplete frame; wherein, to synthesize the frame, the HMD is configured to rotate or shift the previously received frame according to motion vectors received from the external device with the previously received encoded frame."
To review all of new claims, check out Apple's continuation patent 20210165229. Considering that this is a continuation patent, the timing of such a product to market is unknown at this time.