On Thursday the US Patent & Trademark Office published a patent from Apple that relates to a mixed reality headset that includes a 'scene camera' that's part of a holographic direct retinal projector system that is designed to merge views of the real-world and augmented reality imagery together seamlessly.

Apple notes that simulated environments of virtual reality systems and/or the mixed environments of mixed reality systems may be utilized to provide an interactive user experience for multiple applications, such as applications that add virtual content to a real-time view of the viewer's environment, applications that generate 3D virtual worlds, interacting with virtual training environments, gaming, remotely controlling drones or other mechanical systems, viewing digital media content, interacting with the Internet, exploring virtual landscapes or environments, or the like.

In some embodiments, the images may also be analyzed to determine depth information for the scene. The information obtained from the analysis may, for example, be used to place virtual content in appropriate locations in the mixed view of reality provided by the direct retinal projector system.

To achieve a more accurate representation of the perspective of the user, the scene camera is located on the side of the MR headset and facing the inside surface of the lens.

The lens includes a holographic medium recorded with one or more transmission holograms that diffract a portion of the light from the scene that is directed to the user's eye to the scene camera. In this way, the scene camera captures images of the environment from substantially the same perspective as the user's eye.

Apple's patent FIG. 1 below illustrates a mixed reality (MR) system that includes a lens with projection holograms to redirect light beams from a light engine into a user's eye while also passing direct light from the environment to the user's eye.

Apple's patent FIG. 8 above illustrates an example MR system in which the light engine and scene camera are contained in an on-frame unit.

On Thursday, Patently Apple posted a report titled "Apple continues to refine Hand Tracking for a Future Mixed Reality Headset so that a user could use In-Air Gesturing Commands." In that report we mentioned that due to COVID-19 and the traditional summer period, continuation patents or patent-updates were more common that new to market inventions That report a key HMD patent + 3 additional patents that Apple had updated. Some patents/inventions hold more promise than others and receive priority coverage.

Apple's published patent on Thursday titled "Scene Camera" was in fact a patent update but one of great interest knowing that Apple's top executives starting with CEO Tim Cook believe augmented reality is going to be one of the great next things in computing. So even though it's a patent update, it's still interesting and important to what it's updating.

Apple's mixed reality headset is rumored to be coming our way next year and so every bit of knowledge on the first of second-generation devices is worth noting.

In this case, even though Apple was granted this patent four months ago in March 2021, Apple's published continuation patent on Thursday shows that Apple has already decided to scrap the 20 original patent claims altogether and replaced them with 20 new ones.

Patent claims are the heart of every invention/patent. It's what protects the invention from competitors and patent trolls. Obviously, Apple's engineering and legal teams have concluded that this latest important patent needed to be presented in a different light. Below are a few of the new patent claims published on Thursday.

The number sequence of the patent claims have to continue from 21 to 40 instead of 1-20, because it must "continue" from where the invention was first granted, even though they're scrapped. Below are the top four new patent claims:

Patent Claim #21: "A system, comprising: a headset, comprising: a lens with a plurality of layers of a holographic medium on at least one surface of or embedded in the lens; a scene camera located on a side of the headset and facing an inside surface of the lens; and a light engine; wherein at least one of the layers of the holographic medium is recorded with transmission holograms that diffract a portion of wavelengths of direct light from a scene to a user's eye to the scene camera so that the scene camera views the scene from substantially a same perspective as the user's eye views the scene through the lens; wherein at least one layer of the holographic medium is recorded with point-to-point projection holograms; wherein the light engine emits light beams to the projection holograms; and wherein the projection holograms redirect the light beams received from the light engine to an eye box corresponding to the user's eye."

Patent Claim #28: "The system as recited in claim 21, wherein the scene camera is configured to: capture images of the scene from substantially the same perspective as the user's eye views the scene through the lens; and provide the captured images to a controller for the headset; wherein the controller is configured to: analyze the captured images to determine information about the scene; and use the determined information about the scene to place virtual content in appropriate locations in a mixed view of reality provided by the system.

Patent Claim #34: "A method, comprising: diffracting, by transmission holograms recorded in a holographic film on a lens, a portion of wavelengths of direct light from a scene of a real environment to a user's eye to a scene camera; capturing, by the scene camera, an image of the scene; generating virtual content based at least in part on the image of the scene captured by the scene camera; scanning, by a light engine, light beams for the virtual content to projection holograms recorded in a holographic film on the lens; and redirecting, by the projection holograms, the light beams from the light engine to an eye box corresponding to the user's eye to form a mixed reality view that includes the virtual content placed appropriately in the user's view of the real environment as viewed through the lens."

Patent Claim #40: "An optical component for a mixed reality (MR) system, comprising: a lens with a plurality of layers of a holographic medium on at least one surface of or embedded in the lens, wherein the plurality of layers include: at least one layer recorded with transmission holograms that diffract a portion of wavelengths of direct light from a scene to a user's eye to a scene camera so that the scene camera views the scene from substantially a same perspective as the user's eye views the scene through the lens; and at least one layer recorded with point-to-point projection holograms that redirect light beams received from a light engine to an eye box corresponding to the user's eye."

To review the other 16 patent claims, check out Apple's continuation patent 20210208405.

Lastly, for engineers and lawyers who love to see what Apple is updating on key patents, another patent update published last Thursday is presented below:

Patent title: "Privacy Films for Curved Displays." Apple was granted this patent in April 2021. For professionals that want to keep what they're working on private, Apple has the solution and one designed to also work on curved displays. Apple has updated this patent with 20 new patent claims in continuation patent 20210208313 that you could review here.