Today the U.S. Patent and Trademark Office officially granted Apple a patent that relates to user interfaces for interacting with their future HMD that will use a combination of eye gaze and touch controls along with hand and body gestures and even Siri. 

More specifically, Apple's granted patent covers techniques for interacting with a Head Mounted Device (HMD) using eye gaze. A user use will be able to use their eyes to interact with user interface objects displayed on the HMD display. The techniques provide a more natural and efficient interface by, in some exemplary embodiments, allowing a user to operate the device using primarily eye gazes and eye gestures (e.g., eye movement, blinks, and stares).

The HMD includes sensor(s) configured to detect various types of user inputs, including (but not limited to) eye gestures, hand and body gestures, and voice inputs. In some embodiments, input device includes a controller configured to receive button inputs (e.g., up, down, left, right, enter, etc.).

In Apple's patent FIG. 19C below, we see that the user (#200) can more easily select a particular photo from stack (#1908). In FIG. 19C, photos #1908a-1908e are moved from the table (#1912) and presented upright and spread out in the middle of the field of view of the user. In response to receiving user input (#1910a), the photo (#1908a) in the far-left position is designated (e.g., tentatively selected). The designation of photo is indicated by the focus indicator (#1914), which includes a bold border around photo 1908a. In some embodiments, the focus indicator includes a pointer, cursor, dot, sphere, highlighting, outline, or ghost image that visually identifies the designated object. In some embodiments, HMD device #1900 un-designates photo #1908a and returns photos #1908 to the table in response to receiving further input (e.g., selection of an exit button or liftoff of a touch).

In FIG. 19D above, user input #1910b includes a left-to-right swipe or drag gesture. In some embodiments, user input #1910b could include a press of a directional button or an oral command ("move right"). In response to receiving user input 1910b, the focus indicator #1914 is moved from photo #1908a in accordance with (e.g., in the direction of) user input #1910b to designate photo #1908b.

Other Moving Object Patent Figures

In the patent figures below Apple illustrates how using gaze and touch controls will be used to move items within a virtual environment like paintings/photos and even a coffee mug object.

Later in the patent, Apple notes that the HMD system includes image sensor(s), optionally includes one or more visible light image sensors such as charged coupled device (CCD) sensors, and/or complementary metal-oxide-semiconductor (CMOS) sensors operable to obtain images of physical objects from the real environment.

Image sensor(s) also optionally include one or more infrared (IR) sensor(s), such as a passive IR sensor or an active IR sensor, for detecting infrared light from the real environment. For example, an active IR sensor includes an IR emitter, such as an IR dot emitter, for emitting infrared light into the real environment. Image sensor(s) 108 also optionally include one or more event camera(s) configured to capture movement of physical objects in the real environment.

Image sensor(s) also optionally include one or more depth sensor(s) configured to detect the distance of physical objects from the HMD system. In some embodiments, the HMD uses CCD sensors, event cameras, and depth sensors in combination to detect the physical environment around the HMD.  

While the patent is primarily focused on a future Mixed Reality (MR) headset, the described techniques could also be applied to conventional user interfaces on devices such as desktop computers, laptops, tablets, and smartphones.

For more details, review Apple's granted patent 11,132,162. Some of the inventors listed on this granted patent include:

Tim Oriol: Senior Software Engineering Manager, Technology Development Group

Alexis Palangie: Senior Software Engineer

Ryan Burgoyne: Software Developer. Came to Apple via acquisition of Metaio

Rahul Nair: Prototype Engineer