Today the U.S. Patent and Trademark Office officially published a patent application from Apple that without a doubt is one of the most mysterious devices ever conceived by the company. This 2-inch round device that fits into the palm of your hand is chameleon-like in that it could be assigned to work with specific apps or devices like Apple Music, a calculator, a kitchen appliance, Vision Pro. It could also operate as an accessory for Vision Pro, Mac, iPhone or iPad. Odder yet, the device may include a projection system that projects virtual objects into the physical world like a hologram. While it's a "smart" device, Apple could easily spin this into being an AI device that's goes far beyond the recently released Humane AI pin that doesn't work with an XR Headset or other devices.  

Handheld Electronic Devices With Contextual Input-output Capabilities

Electronic devices that are configured to be held in the hand of a user may be used to gather user input and to provide a user with output. For example, a handheld electronic device may have a small, touch-sensitive display for gathering touch input and providing visual output, one or more microphones for gathering voice input, one or more speakers for providing audio output, and location tracking circuitry for tracking the location of the handheld electronic and/or for tracking the location of external electronic devices. If desired, the handheld electronic device may include haptic output devices for providing haptic output to a user.

The handheld electronic device may serve as a voice-controlled assistant, may serve as a stand-alone electronic device running one or more software applications (e.g., a weather application, a music streaming application, a calculator application, a calendar application, a payment application, etc.), may serve as a remote control for household items such as speakers, lights, kitchen appliances, etc., may serve as an anchor or visual marker in an augmented reality or virtual reality system, may serve as an accessory device for gathering input and/or providing output associated with an external electronic device such as a pair of headphones, a loudspeaker, a cellular telephone, a tablet computer, a laptop computer, a head-mounted device, etc., and/or may be used for other functions.

Due to its small size (e.g., smaller than a user's palm, the size of a pebble or stone, the size of coin, or other suitable size) and selective input-output capabilities, the handheld electronic device may be used for focused, contextual applications.

If desired, the handheld electronic device may be used for one application at a time (e.g., a calculator application, a remote control application, a music application, a weather application, a news application, etc.).

Voice commands received by the handheld electronic device may be processed differently depending on the application being used (e.g., control circuitry may listen for calculator-related voice commands when the handheld electronic device is operated in calculator mode, may listen for music-related voice commands when the handheld electronic device is operated in music mode, may listen for weather-related commands when the handheld electronic device is operated in weather mode, etc.).

The mode or application may be set or adjusted based on user input (e.g., voice commands, touch input, etc.), sensor data, and/or location. 

For example, the handheld electronic device may automatically launch into remote control mode when approaching an external electronic device, may automatically launch into contactless payment mode when approaching a payment terminal, may automatically launch into a museum application when entering a museum, etc.

Lastly, when used with Vision Pro in the future, it may display virtual images based on the location of the handheld electronic device. The virtual images may include a live video feed from a video call, a virtual control interface, virtual images associated with voice commands, or other virtual images.

Apple's patent FIG. 2 below is a cross-sectional side view of an all-new illustrative handheld electronic device; FIG. 3 is a diagram of an illustrative system with a handheld electronic device that may be used in conjunction with one or more external electronic devices; FIG. 9 is a diagram of multiple handheld electronic devices providing output based on the relative locations of one another.

Various Use Scenarios

Miscellaneous Points

The housing of device of this mystery device (device #10) may be formed of plastic, glass, ceramics, fiber composites, metal (e.g., stainless steel, aluminum, etc.), fabric, leather, other suitable materials, or a combination of any two or more of these materials. The Housing #58 (FIG. 1) may be formed using a unibody configuration in which some or all of housing is machined or molded as a single structure or may be formed using multiple structures (e.g., an internal frame structure, one or more structures that form exterior housing surfaces, etc.). The Housing may be rigid, may be soft and flexible, or may have rigid and flexible portions.

Shape of Device #10 may have a circular shape, a round shape, an oval shape, a rectangular shape, a pebble or rock shape, a cube shape, and/or other suitable shape. The device may have a lateral dimension D between 25 mm and 50 mm, between 50 mm and 100 mm, between 10 mm and 200 mm, between 5 mm and 75 mm, less than 50 mm, or greater than 50 mm, and may have a thickness T between 0.1 mm and 1 mm, between 0.5 mm and 2 mm, between 1 mm and 2 mm, between 0.1 mm and 5 mm, greater than 5 mm, or less than 5 mm. The form factor of FIG. 2 is merely illustrative. Display 44 may be mounted in housing 58. Display 44 may be round (e.g., circular, oval, etc.), may be rectangular, or may have other suitable shapes.

Attachment Structure: The Device may include an attachment structure for coupling the device to an item such as an external surface, a user's body or clothing, or other item. For example, the device may include an attachment structure such as adhesive (e.g., a semi-permanent adhesive, a skin-safe adhesive, etc.), magnets, clips, hooks, a strap or other band, and/or other structures for attaching the device to an item. This is like the Humane AI Pin Device.

Electrical components #54 (FIG. 1) may include a low-power transmitter (e.g., a Bluetooth® Low Energy transmitter, an ultra-wideband radio-frequency signal transmitter, an RFID transmitter, a near-field communications transmitter, and/or other transmitter) that transmits signals.

Sensors may include force sensors (e.g., strain gauges, capacitive force sensors, resistive force sensors, etc.), audio sensors such as microphones, touch and/or proximity sensors such as capacitive sensors (e.g., two-dimensional touch sensors), optical sensors such as optical sensors that emit and detect light, ultrasonic sensors, and/or other touch sensors and/or proximity sensors, monochromatic and color ambient light sensors, image sensors, fingerprint sensors, temperature sensors, sensors for measuring three-dimensional non-contact gestures (“air gestures”), pressure sensors, sensors for detecting position, orientation, and/or motion (e.g., accelerometers, magnetic sensors such as compass sensors, gyroscopes, and/or inertial measurement units that contain some or all of these sensors), health sensors such as sensors for measuring blood oxygen content and heart rates sensors, radio-frequency sensors (e.g., sensors that gather position information, three-dimensional radio-frequency images, and/or other information using radar principals or other radio-frequency sensing, sensors that detect position, orientation, and/or motion relative to other objects using Bluetooth® positioning techniques or using ultra-wideband positioning techniques), muscle activity sensors (EMG) for detecting finger actions, optical sensors such as self-mixing sensors and light detection and ranging (lidar) sensors that gather time-of-flight measurements, optical sensors such as visual odometry sensors that gather position and/or orientation information using images gathered with digital image sensors in cameras, gaze tracking sensors, visible light and/or infrared cameras having digital image sensors, humidity sensors, moisture sensors, three-dimensional sensors (e.g., time-of-flight image sensors, pairs of two-dimensional image sensors that gather three-dimensional images using binocular vision, three-dimensional structured light sensors that emit an array of infrared light beams or other structured light using arrays of lasers or other light emitters and associated optical components and that capture images of the spots created as the beams illuminate target objects, and/or other three-dimensional image sensors), facial recognition sensors based on three-dimensional image sensors, and/or other sensors, gas sensors, and/or other sensors.

In some arrangements, new device #10 and/or HMD #24 may use sensors and/or other input-output devices to gather user input (e.g., buttons may be used to gather button press input, touch sensors overlapping displays can be used for gathering user touch screen input, touch pads may be used in gathering touch input, microphones may be used for gathering audio input, accelerometers may be used in monitoring when a finger contacts an input surface and may therefore be used to gather finger press input, etc.). If desired, device #10 and/or HMD #24 may include rotating buttons (e.g., a crown mechanism on a watch or finger device or other suitable rotary button that rotates and that optionally can be depressed to select items of interest). Alphanumeric keys and/or other buttons may be included in devices.

Apple filed for this invention in late January 2024 and their engineering team has already filed an update to this invention by adding 20 new patent claims as follows:  

1. 1. A system, comprising: a handheld electronic device having a microphone configured to detect a voice command; and a head-mounted device having a display configured to display an image in response to the voice command, wherein a location of the image is based on a location of the handheld electronic device.
2. 2. The system defined in claim 1 wherein the handheld electronic device comprises a motion sensor that receives gesture input.
3. 3. The system defined in claim 2 wherein the image comprises a virtual control interface that is manipulated using the gesture input.
4. 4. The system defined in claim 1 wherein the image comprises a live video feed from a video call.
5. 5. The system defined in claim 4 wherein the handheld electronic device comprises a camera that captures images during the video call.
6. 6. The system defined in claim 5 wherein the display is configured to display the captured images.
7. 7. The system defined in claim 1 further comprising control circuitry configured to control whether the image is viewable from an additional head-mounted device.
8. 8. The system defined in claim 1 wherein the handheld electronic device comprises a motion sensor that tracks motion of the handheld electronic device and wherein the image moves on the display in response to motion of the handheld electronic device.
9. 9. The system defined in claim 1 wherein the handheld electronic device comprises a visual marker with which the head-mounted device tracks the location of the handheld electronic device.
10. 10. The system defined in claim 9 wherein the visual marker comprises an infrared visual marker and wherein the head-mounted device comprises an infrared camera that detects the infrared marker.
11. 11. A handheld electronic device, comprising: a housing having a lateral dimension smaller than 50 mm; a visual marker on the housing with which a head-mounted device tracks a location of the handheld electronic device; a microphone configured to detect voice input, wherein the head-mounted device is configured to display a virtual control interface in response to the voice input and wherein the virtual control interface is positioned based on the tracked location of the handheld electronic device; and a motion sensor configured to detect gesture input, wherein an item on the virtual control interface is selected in response to the gesture input.
12. 12. The handheld electronic device defined in claim 11 wherein the virtual control interface is for an electronic device and wherein operation of the electronic device is adjusted in response the gesture input.
13. 13. The handheld electronic device defined in claim 12 wherein the electronic device comprises a speaker that plays audio and wherein the audio is adjusted in response to the gesture input.
14. 14. The handheld electronic device defined in claim 11 wherein the visual marker comprises an infrared visual marker.
15. 15. The handheld electronic device defined in claim 11 further comprising a speaker that plays an audio effect in response to the voice input.
16. 16. A head-mounted device, comprising: an infrared camera configured to track a location of a handheld electronic device; a microphone configured to detect voice input; and a display configured to display a virtual image in response to the voice input, wherein the virtual image is anchored to the handheld electronic device.
17. 17. The head-mounted device defined in claim 16 wherein the voice input comprises weather-related voice input and the virtual image comprises a virtual representation of current weather conditions.
18. 18. The head-mounted device defined in claim 17 further comprising a speaker configured to play weather sounds in response to the weather-related voice input.
19. 19. The head-mounted device defined in claim 16 wherein the virtual image comprises a virtual control interface for controlling an electronic device and wherein an item on the virtual control interface is selected using gesture input.
20. 20. The head-mounted device defined in claim 16 wherein a position of the virtual image on the display changes according to changes in the location of the handheld electronic device.

To review the full details of this invention, check out patent application 20240171932.