This week a published patent application from Apple relating to the field of head-mounted displays (HMD) used for applications that immerse in a mixed reality (MR) environment surfaced in Europe. One of the applications discussed in relation to mixed reality headset is chroma keying which is better known generally as Green Screen compositing that is commonly used in movie making and seen nightly on new casts around the world where the actors are news anchors are actually sitting in room with a plain green background and the engineering room fills in the background with relevant content.

In the patent's background they note that chroma keying (or chroma key compositing) is a method for compositing two images by replacing a color range in one image with the content of the second image. It is used in several applications in the field of motion picture or news-casts and weather-casts for instance, often to replace a green (or blue) screen background with a static image or video. An example of green screen compositing is seen below in the making a Superman movie.

While  chroma  keying  is usually  a post-production  effect  or  done  live with  dedicated equipment/software, it can be applied  in the context of  augmented reality (AR) headsets, also referred to as mixed reality (MR), where virtual elements are composited in real-time into the real-world environment seen by the user through the head-mounted display. One example is a crane simulator where the controls would be real objects and where a color screen would be replaced by a virtual crane and a virtual environment. Another example would be a social MR experience using a color screen which covers the whole field of view and only the user's hands and/or body are not replaced by virtual content (unless their color is the same as the screen). This example application is sometimes classified as virtual reality (VR) but for simplicity we here consider it as augmented reality (AR).

Patent Figure 1 presented above illustrates an overview of a non-restrictive illustrative embodiment of the chroma keying setup composed here of a colored screen behind a person. Mixed reality is achieved by having a user look at the setup through a Head-Mounted Display (HMD). The Patent figure is juxtaposed to a photo of a real VRVANA HMD device matching the patent figure.

The overall goals of the invention are to provide a head-mounted device for mixed reality immersive applications; provide a system for mixed reality immersive applications; and provide a method of mixing real environment images with virtual content.

In accordance with Apple's invention, there is provided a head mounted device for mixed reality immersive applications  comprising a processing system composed of a processor, a FPGA and/or a SOC(System on a Chip) and/or GPU for producing virtual content comprising a memory of computer implementable steps; a camera system in communication with the processor system, the camera system providing for capturing real environment images; and a wide-angle lens system positioned in front of a display system in communication with the processing system and with the camera system to show real environment images captured by the camera system mixed and/or augmented with virtual content produced by the processing system thereby providing a mixed reality image; wherein the computer implementable steps comprise: real-time low-latency chroma keying by detecting a user-defined range of color  in the real environment  images captured by the camera system  thereby providing a detected keying masking; compositing the virtual content using the detected keying mask; and displaying the composite virtual content in real time within the mixed reality image via the display system.

Further, in an aspect of the invention there is provided a method of mixing  real  environment   images  with  virtual  content  comprising  reality  immersive applications comprising: producing virtual content; capturing real environment  images; detecting in real time a user-defined range of color in the captured real environment images; providing in real time a detected keying mask; compositing in real time the virtual content using the detected keying mask; mixing in real time the composite virtual content  and the captured real environment  images to  provide a mixed reality image; and displaying the mixed reality image.

The Head-Mounted Display Device

Apple notes that a non-restrictive illustrative embodiment of a Head-Mounted Display system to achieve AR is illustrated in Fig. 2. Two RGB cameras (#34 and #36) and their associated objective lenses (#38 and #40), identified as lens A, capture a stereoscopic view of the environment, similarly to the human visual system #42 and # 44.

In an embodiment, the baseline of the cameras is about 64mm, the average human eye separation (#46), and the position of the sensors need to be aligned as much as possible with the user's eyes to minimize incoherence in the user visual perception.

The optional IR sensors (#19 and #21) and their respective objective lenses (#39 and #41), can be separated by a larger distance to increase the perception of depth estimate by triangulation. 

A user (#1) with an HMD (#33) looks at a display (#48) through wide angle ocular lenses (#50 and #52) identified as lens B. To simulate a pass-through view "X," the images of the cameras are formatted and shown on the display located in front of the eyes, as if the display did not occlude the user's eyes.

The non-restrictive illustrative embodiment considers a single OLED display, but other possible types of displays include two smaller displays, micro-LED, flexible or light-field displays.

The field of view of the camera lenses (#54, 56, 58 and # 60) must closely match the field of view of the eyes (#53, 55, 57 and #59). In this embodiment camera lenses do not induce color aberration on the input rays. The display lenses however do produce color aberration, and thus the viewed color rays (for instance, ray #57) come from different display pixels for each color channel (#61, 62 and #63).

The image processing units that perform image corrections (distortions, chromatic aberrations) and color keying are integrated in a processing board (#65.)

Apple's patent Figure 4A below shows a non-restrictive illustrative embodiment of the chroma keying setup as seen by the cameras of the HMD.

Apple's patent Figure 4B above shows a non-restrictive illustrative embodiment of the chroma keying setup as displayed in the HMD, after the key screen is replaced with a virtual background.

Apple's patent Figure 5A below shows a non-restrictive illustrative embodiment of the chroma keying setup as seen by the cameras of the HMD.

Apple's patent Figure 5B above shows a non-restrictive  illustrative  embodiment  of  the  chroma keying setup as displayed in the HMO, after the key color is replaced with virtual content.

Apple's patent Figure 6 below shows the processing steps done in the HMO in accordance with a non-restrictive illustrative embodiment.

Apple's patent Figure 7 below shows the compositing process done in the HMO in accordance with a non-restrictive illustrative embodiment.

Apple's patent application that was posted in Europe earlier this week was originally filed back in Q3 2017. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Some of the Listed Inventors

It became apparent very quickly when researching the inventors of this patent that none worked at Apple in California even though the company was officially acquired back in 2017.

At one point I thought that perhaps the deal had fallen through because not a single engineer on LinkedIn mentioned Apple. But one key inventor's commentary on his LinkedIn page made it official without ever mentioning Apple. The first inventor listed below provided the hint as follows:

Irv Lustigman: VP Technology & Business Development: 2015-2017 Montreal. Lustigman writes: "Acquired by largest mobile phone/computer manufacturer." The company is still operating as Vrvana out of Montreal and it's pretty clear that they were instructed to not list "Apple" in their linkedIn profiles. Thanks to Mr. Lustigman, he let it out of the bag in his commentary which confirms the 2017 acquisitions and the engineers below being a part of Apple's engineering team.  

Bertrand Nepvue was the CEO and Founder of Vrvana. He left the company in November 2018.  

Simon Fortin-Deschenes: Optical Engineer

Marc-André Chénier: Senior Embedded Software Architect

Yan Cote: CTO and Co-Founder Vrvana

Vincent Chapdelaine-Couture: Computer Vision Scientist. Design and implementation of computer vision methods (realtime feature tracking and depth estimation) to achieve HMD positional tracking and environment tracking in mixed reality applications.

About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.