Today, the U.S. Patent Office published a patent application from Apple  that relates to Techniques for authenticating a user based on hand features. It may be that multiple family members may want to Apple Vision Pro and the system has to recognize the different hands that will be making gestures to control visionOS. In another example, Apple describes a workplace scenario where multiple people collaborating on project with headsets on will be either be given access to only view a project, such as a draft, or permission to edit it based on Vision Pro having registered different people's hands in the office or just on a single project.

Apple notes in their patent background that in Apple Vision Pro, a subset of a person's physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that comports with realistic properties. Some XR environments allow multiple users to interact with virtual objects or with each other within the XR environment. For example, users may use gestures to interact with components of the XR environment. However, what is needed is an improved technique to manage determining identity of a user performing a gesture.

Confirm Gesture Identity

Apple's invention covers sensor data capturing a surrounding environment that may be used to extract hand features. In some embodiments, a 2D image frame may be used to extract hand features, which may be used for hand tracking. According to one or more embodiments, a network may be trained to read in sensor data, such as a 2D image frame, and generate hand features, such as a bounding box, hand keypoints, a hand center, and chirality of the hand. The network may be trained to provide 3D data for the hand features based on a single 2D image frame. For example, the keypoints, bounding box, and/or hand center may be provided in the form of 3D coordinates in space.

In some embodiments, the hand features may be utilized to predict whether a particular hand in the environment belongs to a particular user who should be tracked. For example, a user of a device capturing the sensor data may be tracked, whereas other hands in the scene should be ignored. The hand features for the hands to be tracked may then be utilized for hand tracking techniques.

Apple's patent FIG. 10 below shows, in block diagram form, exemplary systems for confirming gesture identity; FIG. 1 shows a system setup for managing identities of users active in a scene. 

Apple's patent FIG. 2 below shows a flowchart of a technique for registering hands of a user; FIG. 5 shows a system setup for determining authorization of a user based on hand features; and FIG. 8 shows a flow diagram of a technique for extracting hand features from image data.

More specifically to FIG. 5 we see a view #500 of a representation of a multiuser communication session in which several users are participating. The multiuser communication session includes virtual object #510, which is a draft presentation visible to multiple users of the multi-user communication session. As shown, the right hand of user Emily Smith #504 is performing a two finger swipe gesture #514 over the virtual object. Similarly, the right hand of John Doe #508 is also performing a two finger swipe gesture #518 over the virtual object. For purposes of the example, the two finger swipe gesture may be associated with deleting or otherwise manipulating a virtual object.

Apple's patent FIG. 5 also includes authorization list #520. An authorization list may be stored, for example, as an authorization store. In some embodiments, authorization list may be part of a user list managed by an authentication module and/or tracking module.

As depicted, the authorization list includes authorization information associated with objects in the multi-user communication session. As shown in the example authorization list, the presentation (e.g., virtual object #510) may be viewed by John Doe, but not edited. Meanwhile, user Emily Smith is authorized to edit the presentation.

Further, the Anon 1 profile may have no authorization. In some embodiments, anonymous profiles may be associated with no authorization or limited authorization. As such, in some embodiments, the draft presentation may be obfuscated to the user associated with the Anon 1 profile. For instance, the draft presentation virtual objects may not be visible at all to unauthorized users, such as the user associated with the Anon 1 profile. Accordingly, as described above, the gesture #514 associated with the right hand of Emily Smith 504 may cause the virtual object to be manipulated, whereas the gesture #518 associated with John Doe's right hand #508 may be detected, determined to be unauthorized for editing tasks, and ignored.

In some embodiments, a log entry or notification may be generated and/or transmitted related to the unauthorized attempted action by John Doe. Accordingly, these techniques allow the electronic device to confirm the identity of hands performing a particular gesture to prevent unauthorized user within the view of an electronic device from hijacking the device, or otherwise inadvertently causing the device to perform an action based on the gesture.

For finer details about this invention,  review patent application 20230306097.