Today the US Patent & Trademark Office published Apple's first industrial Augmented Reality camera system that relates to a method of and system for projecting digital information onto real objects in a real environment. Apple describes projective AR applications that could be used for prototyping architectural design, car manufacture and more. Whether Apple will be able to use this invention in context to Project Titan is unknown at this time.

In Apple's patent background they note that augmented reality (AR) systems could enhance a real environment by directly visually augmenting the real environment by computer-generated digital information. For example, such digital information is virtual information for augmenting visual impressions of the real environment. Typical applications are known as, for example, so-called projector-based AR, projective AR or spatial AR.

The digital information can be any type of visually perceivable data such as objects, texts, drawings, videos, or their combination. The direct augmentation of the real environment could be realized by projecting the computer-generated digital information onto a surface of a real object of the real environment or a part of the real environment using a projector.

Projective AR has many applications, such as prototyping for architecture design; car manufacture; ubiquitous computing for novel computer user interface development; and information displaying. It's also known in the industry as "Projection or Spatial Augmented Reality."

In order to have a desired alignment between projected visual digital information and real objects that will be augmented in the real environment, spatial transformations between the real objects and a projector that projects the digital information have to be known. For this, a calibration procedure is often performed to estimate the spatial transformation based on 2D-2D, 2D-3D, or 3D-3D correspondences which is a challenging step for building up projective AR systems. A camera is generally required for such calibration procedure.

Many calibration methods have been proposed and developed to compute a spatial transformation between a projector and a real object. One example describes a method to display visual data about a car on a car's surface using a camera and a projector.

The camera and the projector should be rigidly coupled in order to calibrate them once and then assume a projector-car extrinsic (i.e. relative spatial position and orientation) computed from processing visual data acquired by the camera.

The camera detects visual markers attached to the car in order to estimate a spatial transformation between the camera and the car. In this way, a spatial transformation between the projector and the car could be determined. A problem of the approach is that the position and orientation of the markers relative to the car's coordinate system has to be measured beforehand.

Extend3D is described as a commercial projective AR system which tracks a set of markers based on camera sensors that are rigidly attached to a projector.

System for Projecting Digital Information on a Real Object in a Real Environment

Apple's invention covers a method of projecting digital information on a real object in a real environment, comprising projecting digital information on a real object or part of a real object with a visible light projector, capturing at least one image of the real object with the projected digital information using a camera, providing a depth sensor registered with the camera, the depth sensor capturing depth data of the real object or part of the real object, and calculating a spatial transformation between the visible light projector and the real object based on the at least one image and the depth data.

According to another aspect, there is provided a system for projecting digital information on a real object in a real environment, comprising a visible light projector adapted for projecting digital information on a real object or part of a real object in a real environment, a camera adapted for capturing at least one image of the real object with the projected digital information, a depth sensor registered with the camera and adapted for capturing depth data of the real object or part of the real object, and a processing unit arranged for calculating a spatial transformation between the visible light projector and the real object based on the at least one image and the depth data.

According to a particular implementation, in a preferred embodiment the method comprises estimating a spatial transformation between a RGB-D camera system and the real object based on a known 3D model of the real object and computing intrinsic parameters of the projector and a spatial transformation between the projector and the real object based on projecting one or more visual patterns on the surface or surfaces of the real object or a part of the real object using the projector, and capturing a depth map of the projected visual patterns using the RGB-D camera system.

Apple's patent FIG. 1A below shows an exemplary system setup according to an aspect of the invention, wherein in a RGB-D camera system and a visible light projector are not rigidly coupled with each other; FIG. 3 shows a flowchart diagram of a method according to an embodiment of the invention. 

.

Apple's patent FIG. 5 below shows an exemplary hand-held device including a visible light projector and an RGB-D camera system; FIG. 6 shows an exemplary head-mounted device including a visible light projector and an RGB-D camera system.

Apple's patent FIG. 7 below shows an advantageous hardware-setup according to aspects of the invention.

For finer details, review Apple's patent application number 20210076014.

One of the inventors listed on the patent is Selim BenHimane, Senior Engineering Manager, Computer Vision Machine Learning who previously worked at Intel. Another inventor listed is Senior Machine Learning Manager Daniel Kurz who came to Apple when Metaio was acquired. The patent appears to have come from Metaio originally.

Considering that this is a patent application, the timing of such a product to market is unknown at this time.