Yesterday the US Patent & Trademark Office published a patent application from Apple that relates to new input devices that could be used in conjunction with either a full mixed Reality Headset (HMD) or smartglasses to assist the user navigate virtual or augmented environments. The input devices range from wearable finger styled accessories that Apple has filed patents on before and new ones.

Apple's invention covers various kinds of potential handheld controllers to be used when wearing a head-mounted device such as an HMD or smartglasses that will assist the user interact with visual content.

In some embodiments, portions of the input-device (Accessory) may receive some of the user's fingers while leaving other fingers free to move and touch real-world objects.

Apple further notes that during use of the electronic device, the electronic device may gather user housing motion information indicative of changes in position and orientation of the housing by the user.

The device may use an accelerometer or other position sensor mounted in the housing to monitor housing motion. Input-output devices such as optical sensors and other sensors may be used to gather information on the position of the user's fingers as the user interacts with real-world objects.

For example, the tips of the user's fingers may touch tabletops and other real-world surfaces. Other information such as the position of real-world objects in the user's environment may also be gathered using the sensors. The sensors may gather finger position information and other information in real time as the position sensor is detecting housing motion.

In some configurations, the input devices may have multiple structures that move relative to each other such as first and second housing portions. Sensors may be mounted in the second portion of the housing.

When it is desired to monitor finger movement, the second housing portion may be moved to an extended position. This may provide sensors with an enhanced view of the user's fingers, tabletops, and other real-world objects and may otherwise facilitate the gathering of sensor information as the user interacts with real-world objects.

During operation, the system may use the electronic devices to gather input from the user to control the content displayed for the user. For example, user input such as user-induced housing motion input may be gathered from a handheld controller that directs the system to move an item being displayed on the display. Haptic output, audio output, visual output, and other output may be provided to the user in synchronization with the content being supplied on the display.

Apple's patent FIG. 1 below is a schematic diagram of an illustrative electronic device (iPhone, iPad, Macs, glasses, HMD, television +); FIG. 3 is a side view of an illustrative finger device on a user's finger; FIG. 4 is a diagram of an illustrative electronic device with a housing unit coupled to a band; FIG. 5 is a perspective view of an illustrative cylindrical handheld controller with groves configured to receive a finger device for storage.  

Apple's patent FIG. 6 above is a perspective view of an illustrative handheld controller system in which a first handheld controller is stored within a cavity in a second handheld controller; FIG. 7 is a side view of an illustrative handheld controller being used to monitor a user's finger interactions with a surface; FIG. 8 is a front view of an illustrative electronic device having a portion configured to receive a handheld controller device or other equipment for storage.

Further to FIG. 7 Apple states that the accessory may be a handheld device that allows the user's fingers to interact with the surface of an external object (object 34) like a tabletop. As indicated by arrows #42 and #44, sensor measurements (optical, radio-frequency, acoustic, etc.) may be performed using a direct line-of-sight path (arrow #42) or may exploit one or more reflections from surfaces such as the surface of a tabletop.

In addition to using the sensors (#18) to monitor the activities of the tips of fingers (#30), that accessory may use sensors to gather information on the locations of real-world objects and to gather user input associated with contact with the surface of device 10 (e.g., using force sensors, touch sensors, etc.) and/or to gather information on position, orientation, and movement of the accessory device using an inertial measurement unit or other position sensor.

Apple's patent FIGS. 9 and 10 below are perspective views of illustrative handheld controllers with finger openings; FIG. 11 is a perspective view of an illustrative expandable handheld controller; FIG. 12 is a diagram showing how an electronic device such as a handheld controller may have a housing that can be expanded by moving a housing portion into an extended position relative to another housing portion.

Apple's patent FIG. 13 above is a cross-sectional side view of an Apple Watch with joined housing portions and a wrist band; FIG. 14 is a cross-sectional side view of an illustrative device with a wrist band and other housing portions that may be used to house sensors for gathering user input; FIG. 15 is a diagram of an Apple Pencil being used to track the user's environment and a finger of a user as the finger interacts with a surface or other real-world object; FIG. 17 is a perspective view of an illustrative handheld controller with telescoping extensions; FIG. 18 is a perspective view of an illustrative handheld controller.

Apple's patent application 20200356162 that was published today by the U.S. Patent Office was filed back n Q1 2020 supported by a provisional patent filed in Q2 2019. Considering that this is a patent application, the timing of such a product to market is unknown at this time.