On Thursday the US Patent & Trademark Office published a patent application from Apple that relates to advancing Face ID so as to allow for partial face recognition and allowing Face ID to work better under poor lighting situations. All three of Apple's inventors are part of the company's Machine Learning team.

The Problem with Current Face ID

Apple's engineers state that while facial recognition processes may generally be used to identify individuals in an image, the process is often limited in detecting faces in only certain situations.

For example, current face detection processes typically only detect faces in certain orientations (e.g., upright (normal) portrait or landscape modes). Images may often be rotated based on other sensor data to provide upright pictures for face detection, which can be unreliable and processor intensive.

Current face detection processes also typically reject an image for face detection (and downstream processes) if only part of a face is detected in the image. Such images are often rejected because the face detection is not reliable in detecting partial faces.

Face detection processes are also often limited in providing face detection in challenging lighting conditions (low light and/or bright light conditions). The distance between the face of the user and the camera may also adversely affect the effectiveness of the face detection process.

Apple's Face ID Invention / Solution

Apple's invention covers a neural network on a device that implements Face ID, which is a face detection process that captures an image using the TrueDepth camera on the device.

The face detection process may assess if a face is in the image and, if a face is detected, provide a bounding box for the face in the image. The advanced face detection process may provide face detection for any orientation of the face in the image (e.g., the face is detected regardless of the orientation of the face in the image).

Additionally, the face detection process may provide face detection for images that include either the user's entire face or only a portion of the user's face. The bounding box may also be located for an entire face or only the partial face present in the image.

While Apple's patent background includes the need for a solution for Face ID in poor lighting conditions, officially it's only listed in one patent claim stating "wherein at least one image is captured in challenging lighting conditions." Beyond that one mention, there's nothing detailing their solution. Perhaps there will be a second patent / follow-up patent that will focus more on that particular issue.

The rest of the patent focuses on the solution for having Face ID work with partial face recognition and in varying orientations.

Apple's patent FIG. 3 below depicts a representation of an embodiment of a processor on a device; FIG. 4 depicts a representation of an embodiment of a neural network module; FIG. 5 depicts a flowchart for an embodiment of a training process for a neural network module.

Apple's patent FIG. 7 above depicts examples of faces in training images in different orientations; FIG. 8 depicts examples of multiple faces presented to a training process with different partial portions for each of the faces.

Apple's patent FIG. 10 below depicts an example of a bounding box formed (e.g., placed) around a face in image input; FIG. 13 depicts an example of a partial face detected in image input.

Apple's patent FIG. 11 above depicts a flowchart for an embodiment of a test process for a neural network module.

Apple's patent application 20190370529 that was published on Thursday by the U.S. Patent Office was filed back in Q3 2018. Patently Apple is first to report on this invention. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Apple's Inventors

Thorsten Gernoth: Computer Vision Engineering Manager

Feng Tang: Machine Learning Algorithm Manager; Led team developed deep learning and system algorithms for Face ID.

Atulit Kumar: Senior Computer Vision Engineer