Patent wise, Apple's #1 project in 2020 and 2021 has clearly been centered on their future Mixed Reality Headset. Today the US Patent & Trademark Office published a granted patent for Apple that relates to 3D modeling of fingers to be used in a Mixed Reality Headset (not glasses) for object and gesture recognition.

Generating a 3D Model of a Fingertip for Visual Touch Detection

Apple's patent in-part describes generating a 3D model of fingers that could be used for touching virtual objects and for in-air gesturing for the headset to understand commands.

Apple notes in patent point #31 that "a user may spin a finger in front of a camera in order interact with a physical environment or to sign into an account" and so forth.

Apple's patent FIG. 4 presents a flow chart of an example technique for utilizing a 3D model of a fingertip for gesture recognition; FIG. 5 shows an example system diagram illustrating a system setup for generating a 3D model of a fingertip.

Further to FIG. 4, The hand model may include 3D models of a hand corresponding to various poses of the hand. The flowchart continues at #410, and the electronic device 100 (iPhone) updates the 3D model of the hand based on the first and second image. For example, a depth image may be obtained from a perspective such that a touch point between the touching object and the surface is occluded by the touching object (e.g., the fingertip touching a surface may be occluded by the hand from the point of view of the user).

The model of the touching object may be generated or refined based on the combined data from the touch images and/or depth data. The additional information regarding the finger may be utilized to update or refine a 3D model of a hand that includes the finger.

In one or more embodiments, ICP (iterative closet point) algorithms, contour alignment-based algorithms or any algorithm based on mutual information could be used to fit the hand model or finger model with an image of the hand or finger to improve or enable hand tracking or touch detection.

Apple's patent covers both recognizing finger gestures and visual touch detection.

For more details, review Apple's granted patent 10,970,922.