Apple has Won a Patent for a VR Headset and/or Smartglasses that work with a Direct Retinal Projector
Today the U.S. Patent and Trademark Office officially granted Apple a patent that relates to a VR Headset and/or smartglasses that uses a direct retinal projector system to resolve convergence-accommodation conflict. Apple has been working on this for a few years now. Apple was granted another patent related to this project back in September 2021. One of Apple's first in-depth patent application covering a direct retinal projector dates back to September 2019. Today's patent adds to this body of work on a direct retinal projector.
In Apple's patent background they note that conventional virtual reality and augmented reality systems may suffer from accommodation-convergence mismatch problems that cause eyestrain, headaches, and/or nausea. Accommodation-convergence mismatch arises when a VR or AR system effectively confuses the brain of a user by generating scene content that does not match the depth expected by the brain based on the stereo convergence of the two eyes of the user.
For example, in a stereoscopic system the images displayed to the user may trick the eye(s) into focusing at a far distance while an image is physically being displayed at a closer distance. In other words, the eyes may be attempting to focus on a different image plane or focal depth compared to the focal depth of the projected image, thereby leading to eyestrain and/or increasing mental stress.
Accommodation-convergence mismatch problems are undesirable and may distract users or otherwise detract from their enjoyment and endurance levels (i.e., tolerance) of virtual reality or augmented reality environments.
Apple's granted patent covers various embodiments of methods and apparatus for providing virtual reality (VR) or augmented reality (AR) systems. Embodiments of a direct retinal projector are described that may, for example, resolve the convergence-accommodation conflict in head-mounted AR and VR systems. A VR or AR headset system is described that may include or implement different techniques and components of the direct retinal projector.
Embodiments of a gaze tracking component or system are described relating to a direct retinal projector system that tracks the position of a subject's pupil and automatically adjust projection of a scanned light field generated by a projector component of the system so that the scanned light field from the projector enters the subject's pupil.
In some embodiments of a gaze tracking component, a light source (e.g., an infrared (IR) LED) projects a beam of IR light. One or more beam splitters help ensure that the IR beam is roughly q to the center of the scanned light field generated by the projector component of the direct retinal projector system.
Both the projected light and the IR beam are reflected off a 2D scanning mirror and the curved ellipsoid mirror before reaching the subject's eye. A portion of the IR beam entering the pupil of the eye reflects off the retina and emerges from the pupil again (forming a "bright pupil"). The returning IR beam reflects off the curved ellipsoid mirror, the 2D scanning mirror, and the beam splitters to reach a position sensing detector (PSD), for example a quadrant cell technology PSD (also referred to as a quad cell PSD).
A control loop adjusts the 2D scanning mirror to substantially center the returning IR beam on the PSD. In so doing, the 2D scanning mirror is correctly positioned so that the scanned light field from the projector enters the subject's pupil.
Apple's patent FIG. 4A below illustrates a curved, substantially ellipsoid mirror; FIG. 4B illustrates light from a curved ellipsoid mirror of a direct retinal projector striking the pupil at different positions.
Apple's patent FIGS. 5A and 5B below illustrate principle of operation of a gaze tracking system; FIG. 6 illustrates a direct retinal projector that includes a gaze tracking system.
Apple's patent FIG. 9 below illustrates a direct retinal projector that includes an adjustable focusing lens. The projector #2004 of FIG. 9 may, for example, be used in a direct retinal projector system as shown in FIG. 6.
Apple's patent FIG. 11A above illustrates an example optical actuator #3020 that may be used as an adjustable focusing element for the beam in the projector.
One example of a system that is configured to implement any or all of the techniques described herein is illustrated in FIG. 12 below. For example, system 400 illustrated in FIG. 12 may be configured as a virtual reality headset.
Apple's patent FIG. 17 below is a logical block diagram of a frame for a VR/AR device. The figure presents an example of a system (#900) including a frame (#905) that may be configured to hold various elements of a VR/AR device, such as the elements of system #400 of FIG. 12. In various embodiments, the frame may be a glasses frame, a goggles frame, a helmet, or the like, configured to be worn on or over a subject head so as to position the curved mirrors (#415A and 415B) in front of the subject's (#990) left and right eyes, respectively.
For more details, review Apple's granted patent 11,157,072