Today the US Patent & Trademark Office published a series of four patent applications from Apple that relate to performing hand gesture activation on actionable items when wearing an HMD

The first in a series of four patents, Apple's patent titled "Hand-Gesture Activation Of Actionable Items" relates to a method that includes receiving, from the image sensor, one or more images of a physical environment. The method includes detecting, in the one or more images of the physical environment, one or more actionable items respectively associated with one or more actions. The method includes detecting, in the one or more images of the physical environment, a hand gesture indicating a particular actionable item. The method includes in response to detecting the hand gesture, performing an action associated with the particular actionable item.

Apple's patent FIG. 1 below illustrates a perspective view of a head-mounted display device that includes an outward-facing imaging system #170; an eye tracker #182; a scene tracker #190; and earpieces/speakers #152; FIG. 3 illustrates various field-of-views in accordance with some implementations.

Apple's patent FIGS. 4A-4B above illustrate a gaze location indicator #499 that indicates a gaze location of the user, e.g., where in the XR environment #400 the user is looking.

More specifically in patent FIG. 4B, we're able to the XR environment during a second time period subsequent to the first time period. In FIG. 4B, in response to detecting a plurality of actionable items associated with plurality of respective actions, the XR environment includes a respective plurality of glints 471A-471N. Each of the plurality of glints 471A-471N indicates the detection of an actionable item in the physical environment 401.

In FIG. 5a we see that one of the glints represents an actionable indicator. In this case it's a restaurants phone number that the user will be able to point to (touch) to activate a phone call.

A lamp on the desk of the user, for example, could be turned on with a simple tap on the lap with a user's finger when wearing an HMD. The patent describes many examples of how Hand-Gesture Activation of Actionable Items will work with HMDs in the future.

To review its full details, review patent application 20230409122.

Three Additional Patent Applications on the Theme  Covering Varied Features 

• 20230410348: Object Detection Outside A User Field-of-View

• 20230409278: Device With Speaker And Image Sensor
• 20230410458: Occlusion Classification And Feedback