Today the US Patent & Trademark Office published a patent application from Apple that relates to a newly advanced mixed reality headset that addresses "accommodation-convergence mismatch problems" that could cause eyestrain, headaches, and/or nausea in some headsets.

According to Apple, accommodation-convergence mismatch arises when a VR or AR system effectively confuses the brain of a user by generating scene content that does not match the depth expected by the brain based on the stereo convergence of the two eyes of the user.

For example, in a stereoscopic system the images displayed to the user may trick the eye(s) into focusing at a far distance while an image is physically being displayed at a closer distance. In other words, the eyes may be attempting to focus on a different image plane or focal depth compared to the focal depth of the projected image, thereby leading to eyestrain and/or increasing mental stress.

Accommodation-convergence mismatch problems are undesirable and may distract users or otherwise detract from their enjoyment and endurance levels (i.e. tolerance) of virtual reality or augmented reality environments.

Apple's invention covers various embodiments of an augmented reality (AR), and/or mixed reality (MR) direct retinal projector system are described that may, for example, resolve the convergence-accommodation conflict in head-mounted AR, MR, and VR systems.

While at least some embodiments may provide mixed reality, for simplicity the system may generally be referred to herein as an AR system.

Embodiments of an AR headset (e.g., a helmet, goggles, or glasses) are described that may include or implement different techniques and components of the AR system.

In some embodiments, an AR headset may include a reflective holographic combiner to direct light from a projector light engine into the user's eye, while also transmitting light from the user's environment to thus provide an augmented view of reality.

In some embodiments, the holographic combiner may be recorded with a series of point to point holograms; one projection point interacts with multiple holograms to project light onto multiple eye box points. In some embodiments, the holograms are arranged so that neighboring eye box points are illuminated from different projection points.

In some embodiments, the holographic combiner and light engine may be arranged to separately project light fields with different fields of view and resolution that optimize performance, system complexity and efficiency, so as to match the visual acuity of the eye. In some embodiments, the light engine may include foveal projectors that generally project wider diameter beams over a smaller central field of view, and peripheral projectors that generally project smaller diameter beams over a wider field of view.

In some embodiments, the light engine may include multiple independent light sources (e.g., laser diodes, LEDs, etc.) that can independently project from the different projection points, with a proportion being foveal projectors and a proportion being peripheral projectors.

In some embodiments, the light engine includes two or more two-axis scanning mirrors to scan the light sources; the light sources are appropriately modulated to generate the desired image.

In some embodiments, the light engine includes a series of optical waveguides with holographic or diffractive gratings that move the light from the light sources to generate beams at the appropriate angles and positions to illuminate the scanning mirrors; the light is then directed into additional optical waveguides with holographic film layers recorded with diffraction gratings to expand the projector aperture and to maneuver the light to the projection positions required by the holographic combiner.

In some embodiments, the light engine includes a lens for each projector to focus emitted light beams such that, once reflected off the holographic combiner, the light is substantially collimated again when it enters the subject's eye. The required focal surface may be complicated by the astigmatism of the holographic combiner, but is a curved surface in front of the combiner.

The ideal focal surface is different for different eye box positions, and errors may lead to less collimated output. However, in some embodiments, this can be compensated by reducing the beam diameter for different angles where the errors between the ideal focal surface and the actual best fit focal surface are greatest, which alleviates the problem by increasing the F-number and hence the depth of focus of the beam. In some embodiments, these features may be incorporated into a holographic lens.

In some embodiments, active beam focusing elements may be provided for each projection point. This may reduce or eliminate the need to change beam diameter with angle. This may also enable beams that diverge into the eye to, rather than being collimated, match the beam divergence of the supposed depth of the virtual object(s) being projected by the light engine.

With the methods and apparatus presented above, the AR system may not require extra moving parts or mechanically active elements to compensate for the eye changing position in the eye box or for the changing optical power from the holographic combiner during the scan, which simplifies the system architecture when compared to other direct retinal projector systems.

Apple's patent FIG. 4 below illustrates a direct retinal projector system that uses a curved ellipsoid mirror to direct light from a projector into a subject's eye, while also transmitting light from the environment to the subject's eye.

Apple's patent FIG. 5 above illustrates an augmented reality (AR) system that uses a reflective holographic combiner to direct light from a light engine into a subject's eye, while also transmitting light from the environment to the subject's eye.

Apple's patent FIG. 6 below illustrates an AR headset that includes a reflective holographic combiner to direct light from a light engine into a subject's eye, while also transmitting light from the environment to the subject's eye.

Apple's patent FIG. 7 below illustrates high-level components of an AR system.

Apple's patent FIG. 8 below illustrates foveal and peripheral projectors of a light engine in an AR headset; FIG. 9 illustrates light beams from foveal projectors in an AR system.

Apple's patent FIG. 10 below illustrates light beams from peripheral projectors in an AR system.

Apple's patent FIG. 11 below illustrates foveal and peripheral eye boxes for an AR system; FIG. 23 illustrates foveal projections for a holographic combiner.

Apple's patent application that was published today by the U.S. Patent Office was filed back in Q1 2019. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

The Inventors

Alexander Shpunt: Architect. Shpunt came to Apple via the acquisition of PrimeSense where he was the Chief Technical Officer.  

Richard Topliss: Senior Camera Technology Specialist at Apple

Paul Gelsinger: Display Exploration Engineer

Gregory Thomas: Senior Area Manager

Richard Tsai: Sr. Camera Design Engineer

About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Apple IP 20190285897