Today the U.S. Patent and Trademark Office officially granted Apple a patent that relates to methods, systems and/or techniques for providing refined synthetic data, such as for use when training machine learning models. Apple's patent describes how machine learning could be used in conjunction with 3D motion cameras on devices, to better interpret hand gestures, poses and such more accurately.

Apple's patent FIG. 1 below is a logical block diagram illustrating a system configured to implement one or more of the systems, methods and/or techniques; FIG. 2 illustrates a data distribution gap between a synthetic image and a corresponding real image; FIGS. 9A and 9B illustrates an example depth frame and corresponding synthetic image for one example hand pose dataset.

Apple notes that the Generator (FIG. 1, #120) may be coupled with synthesizer #110, thereby allowing synthetic data (e.g., synthetic images) to be refined, such as to make them more realistic. The resulting refined synthetic data may be used as training data (rather than real data) for other machine learning applications, such as text detection, text classification, font recognition, object detection, object classification, hand pose detection, hand pose estimation, hand pose classification, scene recognition, human pose detection, human pose estimation, and/or human pose classification, according to various embodiments.

More specifically, Apple's FIG. 9A above illustrates, according to one embodiment, an example depth frame and FIG. 9B illustrates a corresponding synthetic image for the hand pose dataset. For instance, in one example embodiment, synthesizer #110 may be configured to generate synthetic hand pose images based on a hand pose dataset including training frames captured by multiple 3D motion capturing cameras (e.g., one frontal and 2 side views). For instance, in one example, the depth frames from the frontal camera may be used.

Apple's patent FIG. 10 above illustrates, according to one embodiment, real image samples from a hand pose dataset.

Apple describes the hand gestures being interpreted for a "general-purpose computer system" like an iMac and "other devices" which could also apply to their future Mixed Reality Headset which is known to provide the multiple 3D motion capturing camera system described above.   

For more details on this highly technical patent, review Apple's granted patent US 11475276 B1. 

Apple has been working on hand-gesturing recognition since at least 2016. Other patents on this subject could be found here: 01, 02 and 03