Today the U.S. Patent and Trademark Office officially granted Apple a patent that relates to a future mixed reality system that includes a head-mounted display (HMD) that provides 3D virtual views of a user's environment augmented with virtual content.

Display System having Sensors

Apple's first HMD related granted patent published today covers embodiments of a mixed reality system that may include a mixed reality device such as a headset, helmet, goggles, or glasses (referred to herein as a head-mounted display (HMD)) that includes a projector mechanism for projecting or displaying frames including left and right images to a user's eyes to thus provide 3D virtual views to the user.

The 3D virtual views may include views of the user's environment augmented with virtual content (e.g., virtual objects, virtual tags, etc.). The mixed reality system may include world-facing sensors that collect information about the user's environment (e.g., video, depth information, lighting information, etc.), and user-facing sensors that collect information about the user (e.g., the user's expressions, eye movement, hand gestures, etc.).

The sensors provide the information as inputs to a controller of the mixed reality system. The controller may render frames including virtual content based at least in part on the inputs from the world and user sensors. The controller may be integrated in the HMD, or alternatively may be implemented at least in part by a device external to the HMD.

The HMD may display the frames generated by the controller to provide a 3D virtual view including the virtual content and a view of the user's environment for viewing by the user.

In some embodiments, the sensors may include one or more cameras that capture high-quality views of the user's environment that may be used to provide the user with a virtual view of their real environment.

In some embodiments, the sensors may include one or more sensors that capture depth or range information for the user's environment. In some embodiments, the sensors may include one or more sensors that may capture information about the user's position, orientation, and motion in the environment.

In some embodiments, the sensors may include one or more cameras that capture lighting information (e.g., direction, color, intensity) in the user's environment that may, for example, be used in rendering (e.g., coloring and/or lighting) content in the virtual view.

In some embodiments, the sensors may include one or more sensors that track position and movement of the user's eyes. In some embodiments, the sensors may include one or more sensors that track position, movement, and gestures of the user's hands, fingers, and/or arms.

In some embodiments, the sensors may include one or more sensors that track expressions of the user's eyebrows/forehead. In some embodiments, the sensors may include one or more sensors that track expressions of the user's mouth/jaw.

Apple's patent FIG. 1 below illustrates a mixed reality system; FIGS. 2A through 2C illustrate world-facing and user-facing sensors of a head-mounted display (HMD).

For more details, review Apple's granted patent 11,217,021.