When Apple introduced Vision Pro on June 5, 2023 Mike Rockwell, VP Technology Development Group pointed out that Vision Pro was protected by over 5,000 patents – as presented in the screenshot below. While it's not practical to post reports on all 5,000+ patents, we will continue to post reports on some the major features of Vision Pro. And one of them is how Apple made it easy to work with visionOS by simply using gestures, especially the pinch gesture as presented below.

(Click on image to Enlarge)

Meta liked it so much that they added gestures to control their future OS for Orion Smartglasses as shown in Meta's marketing materials and during their Keynote.

(Click on image to Enlarge)

Yesterday the U.S. Patent and Trademark Office officially published a patent application from Apple that relates to their Pinch Recognition and Rejection System for Vision Pro that allows users to control the basics of visionOS with just hand-gestures.

Apple notes that some devices can generate and present Extended Reality (XR) Environments. An XR environment may include a wholly or partially simulated environment that people sense and/or interact with via an electronic system. In XR, a subset of a person's physical motions, or representations thereof, are tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that comports with realistic properties. Some XR environments allow multiple users to interact with virtual objects or with each other within the XR environment. For example, users may use gestures to interact with components of the XR environment. However, what is needed is an improved technique to manage tracking of a hand performing the gesture.

In some enhanced reality contexts, image data and/or other sensor data can be used to detect gestures by tracking hand data. For some gestures, such as a pinch, the hand pose information is obtained to determine whether a touch occurs between two fingers, or between two portions of a hand.

A framework is applied to determine intentionality of the touch, intentionality may be determined based on hand tracking data. In some embodiments other considerations may be used, such as gaze information, other peripheral object information, user interface (UI) components, or other contextual information. The framework can then determine whether the input action should be enabled based on the intentionality of the gesture.

In some enhanced reality contexts, image data and/or other sensor data can be used to detect gestures by tracking hand data. User input can fail or be unreliable when part or all of the hand is occluded. When a hand becomes occluded or substantially occluded, a gesture can be “locked” so that the state of the hand prior to the occlusion is used for a given frame. In some embodiments, an amount of occlusion can be determined and considered when identifying a gesture.

The XR environment may include Augmented Reality (AR) content, Mixed Reality (MR) content, Virtual Reality (VR) content, and/or the like. With an XR system, a subset of a person's physical motions, or representations are tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment, are adjusted in a manner that comports with at least one law of physics.

As one example, the XR system may detect head movement and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment.

There are many different types of electronic systems that enable a person to sense and/or interact with various XR environments. Examples include: head-mountable systems, projection-based systems, heads-up displays (HUD), vehicle windshields having integrated display capability, windows having integrated display capability, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones/earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop/laptop computers. A head-mountable system may have one or more speaker(s) and an integrated opaque display.

In some implementations, the transparent or translucent display may be configured to become opaque selectively. Projection-based systems may employ retinal projection technology that projects graphical images onto a person's retina. Projection systems also may be configured to project virtual objects into the physical environment, for example, as a hologram or on a physical surface.

in some embodiments, heuristics can be used based on the hand tracking data to determine whether a touch has occurred, and/or a current touch stage. FIG. 5 below shows a flow diagram of an action network, in accordance with some embodiments, which provides an example machine learning process for determining whether a touch has occurred.

The pipeline #500 begins with a set of frames #502 as input. The frames may be a temporal series of image frames of a hand captured by one or more cameras. The cameras may be individual cameras, stereo cameras, cameras for which the camera exposures have been synchronized, or a combination thereof. The cameras may be situated on a user's electronic device, such as a mobile device or a head mounted device.

The frames may be applied to a pose model #504 which may be trained by a neural network configured to predict a 3D pose #508 of a hand based on a given frame (or set of frames, for example in the case of a stereoscopic camera) for a given time.

Apple's patent FIGS. 7A-B above show an example system setup in which a hand gesture is occluded. In particular, FIGS. 7A-B, show a system setup for a user interacting with a device.

Apple's patent FIGS. 9A-9C above depict an example of a user performing a momentum-sensitive gesture, such as a scroll. For purposes of the example diagrams, the hand is clearly shown. However, it should be understood that according to one embodiment, the flow diagram may occur as at least part of the hand is occluded, such as the fingertips. Further, FIGS. 9A-9C are merely presented as an example and are not intended limit the invention.

To review Apple's full 107 patent points/paragraphs along with their 20 patent claims, check out patent application 20240331447.