Today the U.S. Patent and Trademark Office officially granted Apple a patent that relates to a future Mixed Reality Headset System that eliminates accommodation-convergence mismatch problems that detract from a user's enjoyment and endurance levels (i.e. tolerance) of virtual reality or mixed reality environments.

In addition, earlier this month The Information reported that Apple's Mixed Reality Headset would need to connect to another device such as an iPhone. In today's granted patent, Apple describes their mixed reality headset being connected to a MacBook, iMac, iPad or game controller to take advantage of higher-end processors and GPUs.

According to Apple, conventional VR and MR systems may suffer from accommodation-convergence mismatch problems that cause eyestrain, headaches, and/or nausea.

Accommodation-convergence mismatch arises when a VR or MR system effectively confuses the brain of a user by generating scene content that does not match the depth expected by the brain based on the stereo convergence of the two eyes of the user.

For example, in a stereoscopic system the images displayed to the user may trick the eye(s) into focusing at a far distance while an image is physically being displayed at a closer distance. In other words, the eyes may be attempting to focus on a different image plane or focal depth compared to the focal depth of the projected image, thereby leading to eyestrain and/or increasing mental stress.

Accommodation-convergence mismatch problems are undesirable and may distract users or otherwise detract from their enjoyment and endurance levels (i.e. tolerance) of virtual reality or mixed reality environments. This is the problem that Apple's granted patent sets out to eliminate.

Apple's granted patent covers various embodiments of a mixed reality (MR) direct retinal projector system designed to resolve the convergence-accommodation conflict in head-mounted AR, MR, and VR systems.

Embodiments of an MR headset (e.g., a helmet, goggles, or glasses) are described that may include or implement different techniques and components of the MR system. In some embodiments, an MR headset may include a reflective holographic combiner to direct light from multiple (e.g., three) projectors of a light engine into an eye box corresponding to the user's eye, while also transmitting light from the user's environment to thus provide an augmented or mixed view of reality.

The holographic combiner may be recorded with a series of point-to-point holograms; each projector interacts with multiple holograms to project light onto multiple locations (referred to as eye box points) in the eye box. The holograms may be arranged so that neighboring eye box points are illuminated by different projectors.

In some embodiments, only one projector is active at a given time; when activated, a projector projects light from a corresponding light source (e.g., an RGB laser) to all of its eye box points. However, in some embodiments, more than one projector, or all of the projectors, may be active at the same time.

In some embodiments, the MR headset may include a gaze tracking component implemented according to any of a variety of gaze tracking technologies that may, for example, provide gaze tracking input to the controller so that the light beams projected by the light engine can be adjusted according to the current position of the subject's eyes. For example, different ones of the light sources and projectors may be activated to project light onto different eye box points based on the current position of the subject's eyes.

Apple's patent FIG. 5 below illustrates a mixed reality (MR) system that uses a reflective holographic combiner to direct light from multiple projectors into a subject's eye, while also transmitting light from the environment to the subject's eye. Overall, FIG. 5 illustrates architecture, components, and operation of of a direct retinal projector MR system.

Apple's patent FIG. 6 below illustrates an MR headset that includes a reflective holographic combiner to direct light from a light engine into a subject's eye, while also transmitting light from the environment to the subject's eye.

The controller #504 shown in FIG. 6 above may control operation of the light engine(s) #508. The controller may be integrated in the control box #502, or alternatively may be implemented at least in part by a device (e.g., a personal computer, laptop or notebook computer, smartphone, pad or tablet device, game controller, etc.) coupled to the control box via a wired or wireless (e.g., Bluetooth) connection.

The controller may include one or more of various types of processors, CPUs, image signal processors (ISPs), graphics processing units (GPUs), coder/decoders (codecs), memory, and/or other components.

The controller may, for example, generate virtual content for projection by the light engine(s). Gaze tracking component(s) may be implemented according to any of a variety of gaze tracking technologies, and may provide gaze tracking input to the controller so that projection by the light engine(s) can be adjusted according to current position of the subject's eyes.

Apple's patent FIG. 8 below illustrates an MR system that includes a headset with projectors and a separate control box containing light sources.

(Click on image to Enlarge)

For more details, review Apple's granted patent 11,122,256. The listed inventors include Richard Tsai, Sr. Camera Design Engineer; Paul Gelsinger, Optical Engineer; and Gregory Thomas, Senior Area Manager.

Whether this is a spin-off invention related to their ground breaking N301 prototype worked on by Mike Rockwell and team is unknown at this time.