Apple advances their Mixed Reality HMD that's focused on Technologies that deliver a Quality Immersive Experience
Today the US Patent & Trademark Office published a continuation patent application from Apple that relates to relates to the field of head-mounted displays (HMD) used for applications that immerse a user in a virtual reality (VR) or an augmented/mixed reality (MR) environment.
The quality of immersion is subject to several important factors. For instance, characteristics of the display such as image quality, frame rate, pixel resolution, high dynamic range (HDR), persistence and screen-door effect (i.e., the visible lines between pixels on the screen).
Tracking of various elements is generally recognized as an essential prerequisite for achieving a high-end VR and MR application experience. Among these elements, positional head tracking, user body tracking and environment tracking play a key role in achieving great immersion.
Positional head tracking (referred to as positional tracking from here on), which aims to estimate the position and orientation of the HMD in an environment, has to be both low latency and accurate. The reason for this being that the rendered graphics must closely match the user's head motion in order to produce great immersion in VR and the need to correctly align the virtual content in the real world in MR.
User body tracking estimates the position and orientation of the user's body (in particular, but not limited to hands and fingers) relative to the HMD. It can provide in both VR and MR, a means of user input (e.g. hand gestures) enabling interaction with virtual elements. While some positional tracking methods can be used for hand tracking as well (e.g. an IR camera with an array of LEDs on hand-held controllers), other methods take advantage of a smaller analysis space, typically within one meter from the HMD, to increase the robustness of the hand and finger tracking algorithms.
For instance, close-range Time-of-Flight (ToF) cameras can be integrated with or in the HMD. These cameras can yield a depth map of the hands from which a skeletal model of the hands can be constructed. Another approach uses an IR LED flood light together with cameras to segment out and estimate 3D points on the hands and fingers.
Apple's patent FIG. 1 below is a schematic representation of a user wearing head-mounted display (HMD) provided with several cameras and infrared (IR) emitters; FIG. 2A is a schematic top view of an exemplary embodiment of the optics, display and cameras used to achieve both virtual and mixed reality.
Apple's patent FIG. 4A below shows the front view of a first exemplary embodiment of the HMD device, with two RGB cameras optimized for pass-through purposes (MR) and two IR cameras that provide visual data for tracking; FIG. 5 is a flow diagram of the processing steps to achieve VR with positional and user body tracking; FIG. 8 is a flow diagram of an exemplary process to achieve user body tracking. It also includes gesture recognition.
As a continuation patent, Apple has added new patent claims to the original invention. Below are a few of the major additions published today with emphasis placed on the use of an infrared (IR) projector.
Patent Claim #1: A method comprising: at a head-mounted device (HMD) including non-transitory memory, one or more processors, and a communications interface for communicating with first and second RGB stereo-cameras, an infrared (IR) projector, one or more IR sensors, and a display; obtaining, via the first RGB stereo-camera, a first stereo image of a physical environment; obtaining, via the second RGB stereo-camera, a second stereo image of the physical environment; projecting, via the IR projector, an IR pattern of IR electromagnetic radiation onto the physical environment; obtaining, via the one or more IR sensors, texture information for the physical environment associated with a reflection of the IR pattern; generating a dense depth map by matching each pixel in the first stereo image to a different pixel in the second stereo image based at least in part on the texture information; generating a display image by compositing virtual content with the first and second stereo image based on the dense depth map; and displaying, via the display, the display image.
Patent Claim #3: The method of claim 1, further comprising: generating, via the IR projector, the IR pattern according to one of a random or pseudo-random algorithm.
Patent Claim #8: The method of claim 1, wherein the communications interface of the HMD also communicated with one or more IR flood lights, and the method further comprising: operating the IR projector and the one or more IR flood lights according to a temporal multiplexing scheme.
Patent Claim #9: The method of claim 8, further comprising: modulating at least one of exposure time, sensor gain, and intensity of the IR projector and the one or more IR flood lights in real-time to reduce image saturation and improve dynamic range.
Patent Claim #11: A head-mounted device (HMD) comprising: a communications interface for communicating with first and second RGB stereo-cameras, an infrared (IR) projector, one or more IR sensors, and a display; one or more processors; and a non-transitory memory storing one or more programs, which, when executed by the one or more processors, cause the HMD to: obtain, via the first RGB stereo-camera, a first stereo image of a physical environment; obtain, via the second RGB stereo-camera, a second stereo image of the physical environment; project, via the IR projector, an IR pattern of IR electromagnetic radiation onto the physical environment; obtain, via the one or more IR sensors, texture information for the physical environment associated with a reflection of the IR pattern; generate a dense depth map by matching each pixel in the first stereo image to a different pixel in the second stereo image based at least in part on the texture information; generate a display image by compositing virtual content with the first and second stereo image based on the dense depth map; and display, via the display, the display image.
You could review Apple's continuation patent 20210011289 for the complete set of 20 new patent claims here.