The U.S. Patent and Trademark Office officially published a series of 68 newly granted patents for Apple Inc. today. In this particular report, Apple's granted patent relates to the field of head-mounted displays (HMD) used for applications that immerse a user in a virtual reality (VR) or an augmented/mixed reality (MR) environment. In-part, this patent originated with Vrvana which Apple acquired in 2017.

Apple states in their granted patent that the objective of immersion in a virtual world is to convince a user's mind to perceive a non-physical world as if it were real.

The concept of reality here refers more to the notion of perceptual plausibility rather than representing a real world. In virtual reality (VR), immersion is achieved by displaying computer generated graphics that simulate a visual experience of a real or imagined world.

The quality of immersion is subject to several important factors. For instance, characteristics of the display such as image quality, frame rate, pixel resolution, high dynamic range (HDR), persistence and screen-door effect (i.e., the visible lines between pixels on the screen).

The quality of the immerse experience decreases when the displayed field of view is too narrow or if the various tracking functions are slow and/or inaccurate (leading to disorientation and nausea; otherwise known as simulation sickness).

Immersion is also impacted by the camera system performance such as the image quality (noise, dynamic range, resolution, absence of artifacts) and the coherence between the virtual graphics (3D modeling, textures and lighting) and the pass-through images. In mixed reality (MR), virtual elements are composited in real-time into the real-world environment seen by the user. Physical interaction between the virtual elements and real-world surfaces and objects can be simulated and displayed in real-time.

Tracking of various elements is generally recognized as an essential prerequisite for achieving a high-end VR and MR application experience. Among these elements, positional head tracking, user body tracking and environment tracking play a key role in achieving great immersion.

Positional head tracking (referred to as positional tracking from here on), which aims to estimate the position and orientation of the HMD in an environment, has to be both low latency and accurate. The reason for this being that the rendered graphics must closely match the user's head motion in order to produce great immersion in VR and the need to correctly align the virtual content in the real world in MR.

User body tracking estimates the position and orientation of the user's body (in particular, but not limited to hands and fingers) relative to the HMD. It can provide in both VR and MR, a means of user input (e.g. hand gestures) enabling interaction with virtual elements. While some positional tracking methods can be used for hand tracking as well (e.g. an IR camera with an array of LEDs on hand-held controllers), other methods take advantage of a smaller analysis space, typically within one meter from the HMD, to increase the robustness of the hand and finger tracking algorithms.

For instance, close-range Time-of-Flight (ToF) cameras can be integrated with or in the HMD. These cameras can yield a depth map of the hands from which a skeletal model of the hands can be constructed. Another approach uses an IR LED flood light together with cameras to segment out and estimate 3D points on the hands and fingers.

Apple's patent FIG. 1 below is a schematic representation of a user wearing head-mounted display (HMD) provided with several cameras and infrared (IR) emitters; FIG. 2A is a schematic top view of an exemplary embodiment of the optics, display and cameras used to achieve both virtual and mixed reality.

Apple's patent FIG. 4A below shows the front view of a first exemplary embodiment of the HMD device, with two RGB cameras optimized for pass-through purposes (MR) and two IR cameras that provide visual data for tracking; FIG. 5 is a flow diagram of the processing steps to achieve VR with positional and user body tracking; FIG. 8 is a flow diagram of an exemplary process to achieve user body tracking. It also includes gesture recognition.

For more details, review Apple's granted patent 10,838,206 that was issued today by the U.S. Patent Office. One of the lead inventors is Montreal native Simon Fortin-Deschênes who was an Optical Engineer at Vrvana, a company that Apple acquired  back in 2017. The original Vrvana patent dates back to 2017 coinciding with the date on today's granted patent.