Today the US Patent & Trademark Office published a patent application from Apple that was acquired from a Montreal start-up called Vrvana back in 2017 for a Mixed Reality Display. The patent shows that the six inventors listed on the patent including the CEO and Founder of Vrvana didn't join Apple after the acquisition, according LinkedIn information. While this is now an Apple patent, none of the inventors listed are from Apple. How much or any of this patent will be absorbed by Apple's Glasses team led by Mike Rockwell, is unknown at this time. Below is a video that was produced by Vrvana on their Head Mounted Display Device.
The patent background begins by defining Chroma keying (or chroma key compositing) which is a method for compositing two images by replacing a color range in one image with the content of the second image. It is used in several applications in the field of motion picture or newscasting for instance, often to replace a green (or blue) screen background with a static image or video.
While chroma keying is usually a post-production effect or done live with dedicated equipment/software, it can be applied in the context of augmented reality (AR) headsets, also referred to as mixed reality (MR), where virtual elements are composited in real-time into the real-world environment seen by the user through the head-mounted display.
One example is a crane simulator where the controls would be real objects and where a color screen would be replaced by a virtual crane and a virtual environment.
Another example would be a social MR experience using a color screen which covers the whole field of view and only the user's hands and/or body are not replaced by virtual content (unless their color is the same as the screen). This example application is sometimes classified as virtual reality (VR) but for simplicity we here consider it as augmented reality (AR).
Overall, the invention covers a Head-Mounted Display (HMD) including a camera system, image processing units (SoC and/or FPGA systems) and a display system to achieve low latency chroma keying.
The chroma keying processing which is embedded in the HMD replaces a user-defined color range in the camera images by computer generated content and shows the result on the display system. Thus, the camera sensors, the processing system and the display system are used to achieve the low latency chroma keying which replaces the user-defined color range in the images of the environment by the computer-generated content. Specifically, the processing system detects the user selected color range and make a composite image with the virtual content.
Generally stated the invention provides a method to achieve chroma keying inside an HMD used for MR immersive applications. Among such applications, the disclosure is particularly useful in simulations, gaming, entertainment and training.
The present disclosure provides a wearable immersive head-mounted display (HMD) doing real-time color-keying using embedded stereo RGB cameras to capture the environment and a display system to show real-world images augmented with virtual content. The system runs at a high frame rate (typically 75 fps or more) and it achieves a latency of less than one frame by capturing images and formatting them for display in the HMD itself. Low latency is a key element required to achieve great immersion (often called "Presence").
The color keying processing which is embedded in the HMD detects the selected range of color and mixes in the virtual content coming from the HMD itself or from an external source such as a connected computer. Virtual content can also be composited on top of the color-keyed area to allow mixed-reality content occluding real life objects.
Apple's patent FIG. 2 below illustrates a non-restrictive illustrative embodiment of the optics, display and cameras of the mixed reality HMD; FIG. 3 illustrates a schematic overview of a non-restrictive illustrative embodiment of the data flow of the present system.
Apple's patent FIG. 1 below illustrates an overview of a non-restrictive illustrative embodiment of the chroma keying setup composed here of a colored screen behind a person. Mixed reality is achieved by having a user look at the setup through a Head-Mounted Display (HMD).
Apple's patent FIG. 4A above shows a non-restrictive illustrative embodiment of the chroma keying setup as seen by the cameras of the HMD; FIG. 4B shows a non-restrictive illustrative embodiment of the chroma keying setup as displayed in the HMD, after the key screen is replaced with a virtual background; FIG. 5A shows a non-restrictive illustrative embodiment of the chroma keying setup as seen by the cameras of the HMD; and FIG. 5B shows a non-restrictive illustrative embodiment of the chroma keying setup as displayed in the HMD, after the key color is replaced with virtual content.
Apple's patent application 20200252593 that was published today by the U.S. Patent Office was filed by Apple in February 2020. However, the acquired patent actually dates back to August 2017 or about four months prior to the acquisition discovery. Considering that this is a patent application, the timing of such a product to market is unknown at this time.
Bertrand Nepveu: CEO & Fouder of Vrvana; Nepveu didn't join Apple after the acquisition.
Vincent Chapdelaine-Couture: Computer Vision Scientist; the original patent was filed for under Vrvana that Apple acquired. Vincent didn't join Apple after the acquisition.
Anthony GHANNOUM: CTO and CO-founder at Vrvana Display Image Processing Architect at Vrvana; He didn't join Apple after the acquisition.
Irv Lustigman: VP Technology & Business Development at Vrvana; He didn't join Apple after the acquisition.
Marc-André Chénier: Senior Embedded Software Architect at Vrvana: He didn't join Apple after the acquisition.
Simon Fortin-Deschênes: Optical Engineer at Vrvana; He didn't join Apple after the acquisition.