Two patent applications (01 and 02) from Apple have been filed in the last year pointing to a future Apple Pencil-like device that could be used with a Vision Pro family HMD device. Yesterday, the U.S. Patent Office published another patent that advances this project. Most major projects are never revealed in their entirety under a single patent. Many teams work on different aspects of a project and then they're pooled together to provide leaders of the project with the ability to pick and choose which elements of design will make it to the final product.

In Apple's patent background they note that electronic devices such as computers can be controlled using computer mice and other input accessories. In virtual reality systems, force-feedback gloves can be used to control virtual objects. Cellular telephones may have touch screen displays and vibrators that are used to create haptic feedback in response to touch input. Devices such as these may not be convenient for a user, may be cumbersome or uncomfortable, or may provide inadequate feedback. This is what Apple's latest invention is set out to remedy.

Handheld Controllers with Surface Marking Capabilities

Apple states that electronic devices that are configured to be held in the hand of a user may be used to gather user input and to provide a user with output. For example, electronic devices that are configured to control one or more other electronic devices, which are sometimes referred to as controllers, handheld controllers, or handheld input devices, may be used to gather user input and to supply output.

A handheld controller may, as an example, include an inertial measurement unit with an accelerometer for gathering information on controller motions such as swiping motions, waving motions, writing movements, drawing movements, shaking motions, rotations, etc., and may include wireless communications circuitry for communicating with external equipment such as a head-mounted device, may include tracking features such as active or passive visual markers that can be tracked with an optical sensor in an external electronic device, may include input devices such as touch sensors, force sensors, buttons, knobs, wheels, etc., may include sensors for gathering information on the interactions between the handheld controller, the user's hands interacting with the controller, and the surrounding environment.

The handheld controller may include a haptic output device to provide the user's hands with haptic output and may include other output components such as one or more speakers.

Handheld controllers can be held in one or both of a user's hands. Users can use the handheld controllers to interact with any suitable electronic equipment. For example, a user may use one or more handheld controllers to interact with a virtual reality or mixed reality system (e.g., a head-mounted device with a display), to supply input to a desktop computer, tablet computer, cellular telephone, watch, ear buds, or other accessory, to control household items such as lighting, televisions, thermostats, appliances, etc., or to interact with other electronic equipment.

Handheld controllers may gather user input from a user. The user may use the handheld controllers to control a virtual reality or mixed reality device (e.g., head-mounted equipment such as glasses, goggles, a helmet, or other device with a display).

During operation, the handheld controller may gather user input such as information on interactions between the handheld controller(s) and the surrounding environment (e.g., real-world surfaces, objects, etc.), interactions between a user's fingers or hands and the surrounding environment, and interactions associated with virtual content displayed for a user.

Haptic output may be provided to the user's fingers using the handheld controller. Haptic output may be used, for example, to provide the fingers of a user with a desired sensation (e.g., texture, weight, torque, pushing, pulling, etc.) as the user interacts with real or virtual objects using the handheld controller. Haptic output can also be used to create detents, to provide localized or global haptic feedback in response to user input that is supplied to the handheld controller, and/or to provide other haptic effects.

The user input may be used in controlling visual output on a display (e.g., a head-mounted display, a computer display, etc.). For example, the handheld controller may be used in a surface marking mode in which the handheld controller can be used to mark on any surface while a display is used to display the markings made by the handheld controller on the surface. One or more sensors such as a device position sensor and a force sensor may be located at the tip of the handheld controller for tracking surface markings.

The force sensor may be used to detect contact between the tip and the surface. In response, the device position sensor may be operated in surface marking mode to track the location of the tip relative to the surface. Additional sensors such as motion sensors and cameras in the handheld controller and/or the head-mounted device may be used to measure the tilt of the handheld controller relative to the surface during surface marking mode so that the location information gathered by the device position sensor can be transposed to account for tilt. The device position sensor may be an optical flow sensor, a visual inertial odometry camera, an interferometer, and/or other position sensor.

In the illustrative configuration of FIG. 2 above, the controller device #10 includes touch sensor # 42 that may be formed from an array of capacitive touch sensor electrodes such as electrodes #46 overlapping one or more surfaces of housing #54 such as curved surface C, flat surface F, and/or surfaces on tip portions.

The touch sensor may be configured to detect swipes, taps, multitouch input, squeeze input, and/or other touch input.

If desired, the touch sensor may be illuminated, may overlap a display (e.g., to form a touch-sensitive display region on the controller), may overlap an indicator or textured surface, and/or may otherwise be visually or tangibly distinct from the surrounding non-touch-sensitive portions of the housing.

In addition to or instead of touch sensor, the controller device may include one or more other user input devices such as a mechanical input device such as a pressable button, a rotating knob, a rotating wheel, a rocker switch, a slider, or other mechanical input device, a force sensor such as a strain gauge or other force sensor, an optical sensor such as a proximity sensor, a touch sensor such as a capacitive, acoustic, or optical touch sensor, and/or any other suitable input device for receiving input from a user's hand.

Information gathered using sensors such as force sensor input gathered with a force sensor, motion data gathered with a motion sensor (e.g., pointing input, rotations, etc.), location information indicating the location of controller 10, touch input gathered with a touch sensor, and other user input may be used to control external equipment. For example, control circuitry may send control signals to a device that includes instructions to select a user interface element, instructions to scroll display content, instructions to select a different input function for the controller (e.g., to switch from using the controller device as a drawing or writing implement to using controller as a pointing device or game controller.

Apple's patent was filed in August 2024 and published yesterday March 20, 2025. 

Apple's Listed inventors

• David Dashevsky: Sensing System Engineer
• Tong Chen: Optical Sensing Engineer
• Ian Colahan: Product Design Manage
• Christopher Ewy: Product Design Engineer
• Yuhao (Roy) Pan: Product Design Engineer