Apple's engineering teams have working on future handheld controllers that is Apple Pencil-like that could work to control Vision Pro, future Macs, iDevices, Gaming Devices, TV and more. Thus far, Patently Apple has covered three distinct patents on handheld controllers since 2024 began (01, 02 and 03). Today the U.S. Patent and Trademark Office officially published Apple's fourth patent on a possible future controller system since January.

While simple hand gestures may be fine for operating the basics of visionOS, the proposed controller could be used to write in XR environments, be used to see around 3D objects, provide a user with advanced haptics to assist them in feeling the sensation of pushing or pulling a virtual object, to feel textures and more.

Handheld Controllers with Charging and Storage Systems

Apple notes that their patent covers electronic devices that are configured to be held in the hand of a user to assist in gathering user input and to provide a user with output.

A handheld controller may, as an example, include an inertial measurement unit with an accelerometer for gathering information on controller motions such as swiping motions, waving motions, writing movements, drawing movements, shaking motions, rotations, etc., and may include wireless communications circuitry for communicating with external equipment such as a head-mounted device, may include tracking features such as active or passive visual markers that can be tracked with an optical sensor in an external electronic device, may include input devices such as touch sensors, force sensors, buttons, knobs, wheels, etc., may include sensors for gathering information on the interactions between the handheld controller, the user's hands interacting with the controller, and the surrounding environment.

The handheld controller may include a haptic output device to provide the user's hands with haptic output and may include other output components such as one or more speakers.

One or more handheld controllers may gather user input from a user. The user may use the handheld controllers to control a virtual reality or mixed reality device (e.g., head-mounted equipment such as glasses, goggles, a helmet, or other device with a display).

During operation, the handheld controller may gather user input such as information on interactions between the handheld controller(s) and the surrounding environment, interactions between a user's fingers or hands and the surrounding environment, and interactions associated with virtual content displayed for a user.

The user input may be used in controlling visual output on a display (e.g., a head-mounted display, a computer display, etc.). Corresponding haptic output may be provided to the user's fingers using the handheld controller. Haptic output may be used, for example, to provide the fingers of a user with a desired sensation (e.g., texture, weight, torque, pushing, pulling, etc.) as the user interacts with real or virtual objects using the handheld controller.

Haptic output can also be used to create detents, to provide localized or global haptic feedback in response to user input that is supplied to the handheld controller, and/or to provide other haptic effects.

Handheld controllers can be held in one or both of a user's hands. Users can use the handheld controllers to interact with any suitable electronic equipment. For example, a user may use one or more handheld controllers to interact with a virtual reality or mixed reality system (e.g., a head-mounted device with a display – as in a Vision Pro form of headset), to supply input to a desktop computer, tablet computer, cellular telephone, watch, ear buds, or other accessory, to control household items such as lighting, televisions, thermostats, appliances, etc., or to interact with other electronic equipment.

Further, the handheld controllers may have power receiving circuitry configured to receive power from a power source. The power source may be incorporated into an electronic device such as a wireless charging dock or stick, a battery case, or a head-mounted device. The power source may supply power through terminals that form ohmic contacts with mating terminals in the finger device or may transmit power wirelessly using capacitive coupling or inductive charging arrangements. Magnets may be used to hold and align the elongated shaft of the handheld controller on the power source.

A wireless charging stick may include flat surfaces that mate with flat surfaces on one or more handheld controllers. A wireless charging dock may include a recess defined by walls that conform to the shape of the handheld controller. A battery case may include upper and lower housing portions with one or more recesses for respectively receiving one or more handheld controllers.

Click on the graphic below to read the details of the hand controller system. 

For full details, review Apple's patent application 20240069352.