Microsoft Invents a next-gen 'Depth+Multi-Spectral' Camera for Advanced 3D Facial Recognition to Match Apple's Face ID
When Apple introduced Face ID with iPhone X using their new TrueDepth Camera, they instantly leapfrogged their smartphone competitors that were relying on 2D facial recognition as the biometric standard. Only larger bulkier systems like RealSense or Xbox's Kinect were using 3D depth methods. Apple was able to shrink the technology to actually fit a smartphone.
Apple's TrueDepth camera includes an infrared camera, a flood illuminator, a dot projector and more, as shown in the graphic below. The features in combination captured a true 3D depth image of a face. Face ID blew 2D facial recognition out of the water.
With Microsoft now preparing to re-enter the smartphone market with their Surface Duo device in Q4 2020, they apparently want to be able to market the fact that it offers true 3D facial ID for their Hello feature, like Apple's iPhone does with Face ID.
Can you see Microsoft going out of their way to surprisingly introduce an advanced dual display folding smartphone and then rely on older 2D facial recognition cameras? I highly doubt that.
It just so happens that a new Microsoft patent application that surfaced in the U.S. Patent Office's database yesterday details an all-new camera for future devices that they refer to as a "Depth+Multi-Spectral Camera."
Of course the patent stays clear of stating what specific devices it's being designed for because the filing date predates the announcement of Surface Neo and Surface Duo which changes everything. We now know that Microsoft's new Surface devices will want to showcase a new level of security with 3D depth authentication down to the smallest freckle.
Microsoft's introduction states that a camera is configured to output a test depth+multi-spectral image including a plurality of pixels. Each pixel corresponds to one of the plurality of sensors of a sensor array of the camera and includes at least a depth value and a spectral value for each spectral light sub-band of a plurality of spectral illuminators of the camera.
A face recognition machine is previously trained with a set of labeled training depth+multi-spectral images having a same structure as the test depth+multi-spectral image. The face recognition machine is configured to output a confidence value indicating a likelihood that the test depth+multi-spectral image includes a face.
Where Microsoft's camera could excel over Apple's TrueDepth camera is in range beyond a close-up scan. Then again, it was rumored by Bloomberg's Mark Gurman back in January that Apple was working to advance the range for the TrueDepth camera for applications like augmented reality.
More importantly, the next version of the TrueDepth camera may also be destined for the iPhone's rear camera. Microsoft's latest patent never veers off course. It's a pure utility patent that purposely focuses on a single application: short and longer range facial recognition, period.
Microsoft's patent emphasizes one key component that they refer to as a "convolutional neural network" to recognize a face in a depth+multi-spectral image.
Beyond simply introducing this feature within the description area of the patent, Microsoft locked the "convolutional neural network" into their patent claims. That instantly translates to it being a vital component that they want to protect against copycat OEMs. In patent infringement cases, a legal team can only argue or defend patent claims.
So what is a convolutional neural network? In one segment of the patent filing Microsoft it this way:
"Convolutional kernels of further convolutional layers may recognize higher-level two dimensional (2D) visual features, e.g., shapes and patterns, and more generally spatial arrangements of lower-level visual features. Some layers of the convolutional neural network may accordingly recognize and/or locate 2D visual features of faces (e.g., noses, eyes, lips). Convolutional kernels of further convolutional layers may recognize three-dimensional (3D) visual features, e.g., curvature maps. Some layers of the convolutional neural network may accordingly recognize and/or locate 3D visual features of faces (e.g., forehead, chin). Some layers of the convolutional neural network may accordingly recognize and/or locate different textures."
Further, some layers of the convolutional neural network may accordingly recognize and/or locate spectral features of faces (e.g., freckles, eye color). The spectral features may be invariant to temporal factors (e.g., variations in facial expression) and environmental factors (e.g., ambient lighting)." It would appear to be a form of AI.
The first patent figure presented below covers an overview of the new "Depth+Mult-Spectral Camera." We created an easy to view index laying out the components and elements of the new camera for you to follow.
Microsoft's patent FIG. 5 presented below schematically shows different example outputs of an example face recognition machine; FIG. 6 shows an example facial recognition method.
Like most patents from big tech players, the technology is never limited to a single device. Therefore it's very possible that the patent could also apply to future versions of HoloLens and beyond.
Considering that this a patent application, timing to market and how they decide to market it could materially change at any point in time.
Microsoft's inventors listed on the patent filing have a wide spectrum of professional experience behind them collectively in the fields of computer vision, 3D reconstruction, deep learning, depth filtering along with Minecraft and Azure development.
Clearly the camera is destined for a wide variety of future devices over time. I'm not big on making predictions based on patents, though their 2020 Surface Duo smartphone would seem to be a great starting point for this camera.
The U.S. Patent and Trademark Office published this patent application yesterday that was originally filed on May 29, 2018.