Today the U.S. Patent and Trademark Office officially published a patent application from Apple that relates to  computer systems and, more particularly, to input devices for computer systems. While the controlling device may take on the shape of an Apple Pencil, it may take on a different form factor and be used as controller/wand for an Apple Vision Pro type of device. The controlling device could add different input features like miniature scroll wheel and more.

Computer Systems With Handheld Controllers

A system may include an electronic device such as a head-mounted device and a handheld controller for controlling the electronic device. The head-mounted device or other device may have a display configured to display virtual content that is overlaid onto real-world content.

The controller may be used to gather user input and to supply output. A handheld controller may, as an example, include an inertial measurement unit with an accelerometer for gathering information on controller motions such as swiping motions, waving motions, writing movements, drawing movements, shaking motions, rotations, etc., and may include wireless communications circuitry for communicating with external equipment such as a head-mounted device, may include tracking features such as active or passive visual markers that can be tracked with an optical sensor in an external electronic device, may include input devices such as touch sensors, force sensors, buttons, knobs, wheels, etc., may include sensors for gathering information on the interactions between the handheld controller, the user's hands interacting with the controller, and the surrounding environment. The handheld controller may include a haptic output device to provide the user's hands with haptic output and may include other output components such as one or more speakers.

One or more handheld controllers may gather user input from a user. The user may use the handheld controllers to control a virtual reality or mixed reality device (e.g., head-mounted equipment such as glasses, goggles, a helmet, or other device with a display).

During operation, the handheld controller may gather user input such as information on interactions between the handheld controller(s) and the surrounding environment, interactions between a user's fingers or hands and the surrounding environment, and interactions associated with virtual content displayed for a user.

The user input may be used in controlling visual output on a display (e.g., a head-mounted display, a computer display, etc.). Corresponding haptic output may be provided to the user's fingers using the handheld controller. Haptic output may be used, for example, to provide the fingers of a user with a desired sensation (e.g., texture, weight, torque, pushing, pulling, etc.) as the user interacts with real or virtual objects using the handheld controller. Haptic output can also be used to create detents, to provide localized or global haptic feedback in response to user input that is supplied to the handheld controller, and/or to provide other haptic effects.

Handheld controllers can also be held in one or both of a user's hands. Users can use the handheld controllers to interact with any suitable electronic equipment.

For example, a user may use one or more handheld controllers to interact with a virtual reality or mixed reality system (e.g., a head-mounted device with a display), to supply input to a desktop computer, tablet computer, cellular telephone, watch, ear buds, or other accessory, to control household items such as lighting, televisions, thermostats, appliances, etc., or to interact with other electronic equipment. 

Apple's patent FIG. 2 below is a schematic diagram of an illustrative system with a handheld controller that could be used with an HMD like Vision Pro, an iPhone or iPad, MacBook or desktop Mac, future smartglasses , a gaming device and many applied to many applications.  

Apple's patent FIG. 4 is a perspective view of an illustrative handheld controller with a knob that rotates about a longitudinal axis of the handheld controller; FIG. 5 is a perspective view of an illustrative handheld controller with a knob that is accessible on two opposing sides of the handheld controller and that rotates about an axis that is perpendicular to the longitudinal axis of the handheld controller.

Apple's patent FIG. 6 above is a perspective view of an illustrative handheld controller with a knob that is accessible on one side of the handheld controller and that rotates about an axis that is perpendicular to the longitudinal axis of the handheld controller; FIG. 7 is a perspective view of an illustrative handheld controller having first and second mating housing portions; FIG. 8 is perspective view of the illustrative handheld controller of FIG. 7 in a configuration where the first and second mating housing portions have been separated from one another.

Apple's patent FIG. 9 below is a diagram of an illustrative system including a handheld controller and an electronic device configured to track a location of the handheld controller; FIG. 11 is a top view of an illustrative head-mounted device having support structures configured to support a display and sensors such as a gaze tracker and forward facing camera; FIG. 12 is a side view of an illustrative handheld controller onto which computer-generated display content is overlaid.

Apple's patent FIG. 13 above is a side view of an illustrative handheld controller having a tip portion onto which a computer-generated paint brush head is overlaid; FIG. 14 is a side view of an illustrative handheld controller having a tip portion onto which a computer-generated tool head is overlaid in accordance with an embodiment.

For full details, review Apple's patent application 20240012496.