Today the U.S. Patent and Trademark Office officially published a patent application from Apple that relates to electronic devices with controllers that could move virtual objects in a three-dimensional world, adjust audio settings, draw items in a three-dimensional drawing program, play three-dimensional video games and more on an iPad, Mac, Vision Pro and more.

Systems With Deformable Controllers

Apple's invention covers a controller that may be provided for use with an electronic device. The electronic device may be an iPad, Mac or Vision Pro (a head-mounted device) that gathers user input and that controls visual content based on the user input. It may include an Apple Pencil or other accessory. It may be a handheld device such as a remote control, and/or may be other electronic equipment.

The controller may be mounted on a head-mounted housing or other electronic device housing. A user may provide input to the controller with one or more fingers. The input may be sensed using sensing circuitry in the controller.

The controller may have a nub-shaped housing with an elongated dome shape that extends along an axis. The housing may be formed from a flexible housing structure that allows the housing to be deformed. User finger input may be provided to the controller during operation.

Sensor circuitry in the flexible housing may be used in detecting user finger input such as multitouch touch input (e.g., pinch input, twisting input, sliding gestures, tap input, etc.), force input in which a user's finger presses against the flexible housing, and/or deformation input in which the flexible housing is bent away from the axis or otherwise deformed by a user's finger.

A deformable controller may be provided that allows a user to move virtual objects in a three-dimensional world. The deformable controller may be used, for example, to translate objects in multiple dimensions and/or to rotate objects in three dimensions.

The controller will work with content displayed on an iPhone, iPad, Mac, Vision Pro or smartglasses and may include visual content such as three-dimensional computer-generated content (e.g., visual content such as virtual reality content presented to a user with displays in a head-mounted device), audio content, haptic output, and/or other output.

At the same time, one or more of these same devices in the system may use input-output circuitry to gather user input that is used in interacting with the content. As an example, input-output circuitry #22 may gather user input that is used to make menu selections, move virtual objects in a three-dimensional world, adjust audio settings, draw items in a three-dimensional drawing program, play a three-dimensional video game, and/or otherwise interact with system To facilitate these interactions.

The Input-output circuitry may include sensors such as sensor circuitry for deformable controllers and other sensors. The sensors may include, for example, three-dimensional sensors (e.g., three-dimensional image sensors such as structured light sensors that emit beams of light and that use two-dimensional digital image sensors to gather image data for three-dimensional images from light spots that are produced when a target is illuminated by the beams of light, binocular three-dimensional image sensors that gather three-dimensional images using two or more cameras in a binocular imaging arrangement, three-dimensional lidar sensors, three-dimensional radio-frequency sensors, or other sensors that gather three-dimensional image data), cameras (e.g., infrared and/or visible digital image sensors), gaze tracking sensors (e.g., a gaze tracking system based on an image sensor and, if desired, a light source that emits one or more beams of light that are tracked using the image sensor after reflecting from a user's eyes), strain gauges, touch sensors, capacitive proximity sensors, light-based (optical) proximity sensors, other proximity sensors, force sensors, sensors such as contact sensors based on switches, gas sensors, pressure sensors, moisture sensors, magnetic sensors, audio sensors (microphones), ambient light sensors, microphones for gathering voice commands and other audio input, sensors that are configured to gather information on motion, position, and/or orientation (e.g., accelerometers, gyroscopes, compasses, and/or inertial measurement units that include all of these sensors or a subset of one or two of these sensors), and/or other sensors. If desired, sensors such as these may be used in forming deformable controllers and may be fully or partially housed within a deformable controller housing.

To review the full details of this invention, check out patent application 20240192811.