This week the U.S. Patent and Trademark Office officially granted Apple a patent that was originally filed in April 2020 regarding a possible accessory device for Apple Vision Pro. More specifically, a unique virtual trackpad accessory that users can use in virtual worlds or games to move and/or interact with objects.

Further, a system may have a projection system that projects virtual objects into the physical environment, for example, as a hologram or on a physical surface, so that a person, using the system, perceives the virtual objects superimposed over the physical environment.

Apple first notes that today, techniques for interacting with virtual objects can be cumbersome and non-intuitive for a user. Apple's invention covers techniques for remote touch detection using a system of multiple devices, including a peripheral device that is placed on a physical surface such as the top of a table. With these techniques, a user can interact with virtual objects by performing touches on a physical surface.

The system will use image sensor(s) to track the position and orientation of display(s) relative to one or more fixed objects in the physical environment. In some embodiments, the system uses image sensor(s) that are inward facing (e.g., facing the user) for gaze tracking and/or hand motion tracking, which can be used, for example, to control a user's avatar.

In addition, the system may have a projection system that projects virtual objects into the physical environment, for example, as a hologram or on a physical surface, so that a person, using the system, perceives the virtual objects superimposed over the physical environment. 

Apple's patent FIGS. 1A-1B below depict exemplary systems for use in various CGR technologies; FIG. 2 illustrates an example of a system comprising a head-mounted display device and a peripheral device for enabling remote touch detection.

More specifically, Apple's patent FIG. 2 above illustrates system #200, including peripheral device #200A and head-mounted display (HMD) device #200B.

The peripheral device includes camera sensor(s) #210 (e.g., image sensor(s) #108) and motion sensor(s) (e.g., orientation sensor(s) #110). Additionally, HMD device #200B can itself be an embodiment of system #100.

Apple's system #200, including peripheral device #200A and HMD device #200B, enables accurate remote touch detection on a desktop or general table surface #206 in order to interact with (e.g., control, manipulate, activate, select) displayed UI elements in a CGR environment displayed using HMD device.  

In some embodiments, system #200 forms a three-angle system (e.g., one camera sensor on peripheral device 200A and two camera sensors on the HMD.

In some embodiments, the system #200 includes peripheral device #200A (e.g., with no camera sensors) that emits infrared light to be detected by camera sensor(s) on the HMD.

In some embodiments, system excludes the peripheral device and relies on the HMD device to perform remote touch detection using depth sensing technologies.

Apple's patent FIG. 3 below illustrates an example of a notification that prompts the user to correct an error condition; FIGS. 4-5 illustrate an example of remote touch detection in a CGR environment.

Further, Apple's patent FIG. 3 above illustrates the perspective from which user is viewing CGR environment via a transparent or translucent display of the HMD. In some embodiments, the display of the HMD is opaque, so the user views the physical environment using pass-through video.

Apple's patent FIGS. 6 and 7 below illustrate additional examples of remote touch detection in a CGR environment. Here you're able to see how the user is able to use a virtual trackpad to move the photo sideways to upright using a simple hand gesture.

Apple's patent FIGS. 11-12 above illustrates an alternative example of a system comprising a head-mounted display device and a peripheral device for enabling remote touch detection via infrared emitters instead of cameras to detect objects.

More specifically, Apple notes that "In some embodiments, peripheral device #200A includes infrared emitter(s) #220, which emit infrared light to enable remote touch detection.

The virtual trackpad will allow for traditionally known trackpad gestures such as tap, double-tap, drag, flick, pinch, spread, press, rotate). In some embodiments, the criteria includes a criterion that is met when the input (or movement of the object) is a tap gesture. In some embodiments, the criteria includes a criterion that is met when the input (or movement of the object) is a pinch or spread gesture. In some embodiments, the criteria includes a criterion that is met when the input (or movement of the object) is a rotate gesture. In some embodiments, the criteria includes a criterion that is met when the input (or movement of the object) corresponds to (e.g., was performed by) one or more digits (e.g., finger) of a user. In some embodiments, the set of one or more criteria is not met when the movement of the object is performed by an invalid object (e.g., an object other than a finger or digit of a user).

For finer details, review Apple's granted patent 11714540.

Some of the inventors include Sam Iglesias, Senior Software Engineer, AR/VR; Rohit Sethi, Engineering Manager specializing in deep learning and computer vision; and Lejing Wang, Tech Lead/Engineering Manager.