On February 12, Patently Apple posted a report titled "Apple is developing a way to improve future Hand Tracking Accuracy by using 'Spatiotemporal Smoothing.'" Today the U.S. Patent and Trademark Office officially granted Apple a patent that relates to advancing pinch recognition using finger zones.

In Apple's granted patent background they note that in XR, a subset of a person's physical motions, or representations thereof, are tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that comports with realistic properties. Some XR environments allow multiple users to interact with virtual objects or with each other within the XR environment. For example, users may use gestures to interact with components of the XR environment. However, what is needed is an improved technique to manage tracking of a hand performing the gesture.

Pinch Recognition Using Finger Zones

Apple notes that in some enhanced reality contexts, image data and/or other sensor data can be used to detect gestures by tracking hand data. For some gestures, such as a pinch, the hand pose information is obtained to determine whether a touch occurs between two fingers, or between two portions of a hand.

A framework is applied to determine intentionality of the touch, intentionality may be determined based on hand tracking data. In some embodiments other considerations may be used, such as gaze information, other peripheral object information, user interface (UI) components, or other contextual information. The framework can then determine whether the input action should be enabled based on the intentionality of the gesture.

According to one or more embodiments, particular zones of the hand may be used to determine intentionality of a pinch. For example, predefined zones may be determined on a user's hand or hands. These zones may be located, for example, around a fingertip of a thumb and one or more other fingers.

A touch between two zones may be determined to be an intentional pinch, whereas a touch that includes two portions of the user's hand that are not both in the zones may be determined to be an unintentional touch. In some embodiments, a touch between two portions of a user's hand that are not both within the determined zones may not be classified as a touch, thereby reducing compute by avoiding determining intentionality.

Further, in some embodiments, an input action may be cancelled when a release of the pinch occurs at a different point of contact than the touch. Moreover, by using touch zones, a machine learning classifier trained for pinch recognition becomes more robust by constraining the problem space.

In some embodiments, a determined pinch can initiate an action, which can thereafter be cancelled based on a movement of the user's fingers during the touch. That is, the pinch can begin with a touch between two portions of the user's hand within the zones. The user may maintain contact between two portions of the hand, but one or both of the points of contact may be within the zones. That is, the user's fingers may “slide” out of the zones while maintaining contact. In this situation, the pinch may no longer be considered to be intentional, and an associated user input action associated with the pinch can be cancelled.

To review the full details of this invention, check out granted patent 12229344.