Today the U.S. Patent and Trademark Office officially published a patent application from Apple that relates to techniques for controlling and/or moving a cursor, such as by using air gestures for Vision Pro or future smartglasses or a trackpad with Macs.

Apple's patent application notes that some methods and interfaces for interacting with three-dimensional environments are cumbersome, inefficient, and limited. For example, some systems do not enable use of a cursor to interact with virtual objects, which can make performing precise operations, such as text editing or drawing, cumbersome and/or inefficient. Further, other systems that enable the use of a cursor to interact with virtual objects are complex, tedious, and error-prone, create a significant cognitive burden on a user, and detract from the experience with the virtual/augmented reality environment. In addition, these methods take longer than necessary, thereby wasting energy of the computer system. This latter consideration is particularly important in battery-operated devices.

Accordingly, there is a need for computer systems with improved methods and interfaces for interacting with virtual objects using a cursor to make interaction with three-dimensional environments more efficient and intuitive for a user. Such methods and interfaces optionally complement or replace conventional methods for interacting with three-dimensional environments. Such methods and interfaces reduce the number, extent, and/or nature of the inputs from a user by helping the user to understand the connection between provided inputs and device responses to the inputs, thereby creating a more efficient human-machine interface.

Apple's patent could relate to Vision Pro, future smartglasses, an iPad or Mac.

Apple's patent FIG. 7A below illustrates an example of controlling a cursor based on detected hand movement. Device #700 is implemented as a head-mounted device that includes display #706, that is in communication with one or more image sensors/cameras to detect in-air gestures from the user (or using a large trackpad #708).

Device #700 below displays window #710 for a photos application and window 712 for a markup application (e.g., an application that allows the user to create and/or edit content, e.g., by drawing, typing, and/or adding other types of graphical elements into the content).

Window #710 includes favorites icon #711 and recently added icon #714. The favorites icon is currently selected (as indicated by the bolded border around it). The device thus displays, in the window the user's favorite photos #716, labeled #1-6.

Window #712 includes cursor icon #718 and markup icon #720. Both the cursor icon and markup icon are in an unselected state, thereby respectively indicating that a cursor is not invoked and that a markup mode (e.g., a drawing mode in which the cursor can be used to draw marks on content) for the markup application is inactive. The window further displays text #730 included in the content being created and/or edited using the markup application.

In FIG. 7A, while the cursor icon is in the unselected state, the HMD device detects that the user is currently focused on (e.g., gazing at) cursor icon #718. The HMD Device thus displays focus indicator #722 around cursor icon.

In other embodiments, the HMD device detects that the user's gaze is directed to cursor icon, but doesn't display a focus indicator to indicate focus on a particular user interface element. For example, the HMD device instead changes one or more visual characteristics (e.g., brightness, size, depth, color, border width, text size, and/or font) of a currently in-focus user interface element.

While the HMD device displays the focus indicator around the cursor icon, the HMD detects gesture #724 (e.g., a pinch gesture detected by trackpad #708) performed by right hand that selects cursor icon. In some embodiments, such as embodiments in which the HMD device is a head-mounted device, gesture #724 is an air gesture.

Apple's patent FIG. 7F2 above illustrates an embodiment in which windows #710 and #712 and cursor #726 are displayed on display module #X702 of head-mounted device (HMD) #X700. In some embodiments, device X700 includes a pair of display modules that provide stereoscopic content to different eyes of the same user. In some embodiments, the second display module displays a slightly different image than display module X702 to generate the illusion of stereoscopic depth.

Apple's patent application 20240256049 is long, detailed with many example graphics. For Safari users. For Chrome/Fire/Edge browsers.