Today the US Patent & Trademark Office published a patent application from Apple that relates to computer systems that are in communication with a display generation component and, optionally, an Apple Watch that provides computer-generated experiences, including, but not limited to, electronic devices that provide virtual reality and mixed reality experiences via a display beyond Apple Vision Pro, such as an iPad, MacBook, iMac and a TV via an Apple TV box.

Devices, Methods & GUIs for Interacting with Virtual Objects using Hand Gestures

During the WWDC23 keynote we finally saw the use of in-air hand gestures come to life with the Apple Vision Pro headset in conjunction with eye-tracking. Now that it's a proven methodology used to manipulate what's on a headset display, Apple may extend this methodology to work with other display devices such as a TV (via Apple TV box), iPad, MacBook and Apple Displays for desktops.  

In some embodiments, the functions performed through the interactions optionally include image editing, drawing, presenting, word processing, spreadsheet making, game playing, telephoning, video conferencing, e-mailing, instant messaging, workout support, digital photographing, digital videoing, web browsing, digital music playing, note taking, and/or digital video playing.

Apple's patent FIG. 7A below illustrates an example technique for navigating content on a TV (using Apple TV box) using hand gestures directed by  the rotation of the hand wearing an Apple Watch which is communicating with the Apple TV box. This could of course apply to a future iMac, MacBook and more as well.

As illustrated in FIG. 8H below, in response to receiving an indication that Apple Watch has detected that the user has performed a request to display control center user interface #830 which contains a plurality of virtual objects. Here the user could also perform a second operation such as initiate the playback of a video media item (Apple TV content), modifying the playback status of an audio media item, and/or initiate a photo capturing process.

Apple's patent FIG. 6 below is a flow diagram illustrating a glint-assisted gaze tracking pipeline; and FIG. 12B illustrates a user controlling virtual objects with the Apple Watch and iPad (or MacBook) camera.

For deeper details, review Apple's patent application number 20230252737.