Apple Vision Pro allows a user to control or interact with visionOS with simple hand gestures. Yet there may be times where additional “controllers” may be required to control Apple TV content, video game play, a presentation and more and on other devices such as Macs and smartglasses. Two of today’s patent applications point to users controlling on device content with “finger devices,” an Apple Pencil and more.

Systems Having Peripherals With Magnetic Field Tracking

Apple’s patent relates to a head-mounted device that works with a wireless controller that includes an accelerometer for measuring the orientation of the controller relative to the Earth's gravity and a direct-current magnetometer for measuring the orientation of the controller relative to the Earth's magnetic field.

The wireless controller may determine the position and orientation of the wireless controller relative to the head-mounted device. This information can then be wirelessly transmitted to the head-mounted device to control the head-mounted device. The alternating-current magnetometers may be three-coil magnetometers located at different locations in a controller housing.

Each three-coil alternating-current magnetometer may have three coils that are aligned along three respective axes that are linearly independent (none of the axes are parallel to each other). In an illustrative configuration, which may sometimes be described herein as an example, each three-coil alternating-current magnetometer may have three orthogonal coils.

In some configurations, coils may share a common magnetic core. Coils may also have air cores or separate magnetic cores, if desired. In an illustrative arrangement, two or three coils of each alternating-current magnetometer may be wrapped around a common magnetic core.

The wireless controller may have additional components to help measure position and orientation. These components may include optical sensors such as cameras that form part of a visual inertial odometry system, self-mixing proximity sensors, and/or optical-flow-based visual inertial odometry system sensors. If desired, the wireless controller may have light sources that serve as visual reference points. A camera in a head-mounted device may track these visual reference points to help determine the position and orientation of the wireless controller.

Apple’s patent FIG. 1 below is a diagram of an illustrative system in which a head-mounted device (Apple Vision) is being wirelessly controlled by a peripheral device (sometimes referred to as an accessory or controller). As shown in FIG. 1, system 8 may include a controller such as wireless controller #10B that is used in wirelessly controlling an electronic device such as head-mounted device  #10A. The Wireless controller may be worn on a user's finger, held in a user's hand, or removably coupled to a portable electronic device that is held in a user's hand or otherwise moved by a user. 

As the user moves the controller (and optionally presses buttons, touches a touch sensor, applies force to a force sensor, or otherwise provides input to the controller), the position and orientation of the controller 10B be monitored by sensor circuitry in the controller (and, if desired, a camera or other sensor circuitry in device 10A).

The information that controller gathers on the position, orientation, and movement of the controller serves as user input for the HMD.

The user input from controller may be provided wirelessly to device 10A in real time and used in controlling the operation of the controller. For example, user input supplied by the controller  may control the movement of elements in a game, may move a pointer, may be used in navigating through on-screen menu items and making menu selections, and/or may otherwise be used in controlling the user's interactions with the HMD.

Apple’s patent FIG. 2 below is schematic diagram of an illustrative system with a head-mounted device and controller.

Apple’s patent FIGS. 3-5 illustrate a variety of controller styles that could work with an HMD; FIG. 8 is a perspective view of an illustrative finger device that may serve as a controller – and is the primary controller presented in the patent; and FIG. 10 is a side view of an illustrative finger device having magnetic sensors

Further to fig. 8, Apple notes that magnetic sensors and other sensors #16 may detect the position, orientation, and changes in position and orientation of controller #10B during these free space movements, so that air gesture input may be used in controlling the HMD (device 10A).

To review the invention’s full details, review patent application 20230333394.

Multi-Finger Gesture Based On Finger Manipulation Data And Extremity Tracking Data

A second finger device patent was published last week by the U.S. Patent Office  In accordance with some implementations, a method is performed at an electronic device with one or more processors, a non-transitory memory, a display, an extremity tracking system, and a communication interface provided to communicate with a finger-wearable device. The method includes displaying a computer-generated object on the display. The method includes obtaining finger manipulation data from the finger-wearable device via the communication interface. The method includes determining a multi-finger gesture based on extremity tracking data from the extremity tracking system and the finger manipulation data. The method includes registering an engagement event with respect to the computer-generated object according to the multi-finger gesture.

The finger-wearable device can be worn by a finger of a user. In some implementations, the electronic device tracks the finger with six degrees of freedom (6DOF) based on the finger manipulation data.

Apple’s patent FIG. 3D is one example of an electronic device registering engagement events with respect to a computer-generated object based on respective multi-finger gestures.

In this patent, finger devices could work with iPhone, iPad, a MacBook, iMac and HMDs (including smartglasses)

Review Apple’s patent application 20230333651 for more details.

And lastly, Patently Apple discovered a third Apple patent on this subject matter titled “Gesture Tutorial for a Finger-Wearable Device” under #20230333650.

In this patent, finger devices could work with iPhone, iPad, a MacBook, iMac and HMDs (including smartglasses)

Review Apple’s patent application 20230333651 for more details.