The US Patent & Trademark Office published a patent application from Apple yesterday titled "Electronic Device with Intuitive Control Interface." Apple has invented a display system and circuitry that could work with a head mounted device like a VR headset, eyeglasses or an iPhone. Apple's new system will allow a non-geek to set up a wireless home system like a pro by simply viewing their home devices in a living room, for instance, and connecting them by dragging their finger from one device to another and it's done. In our cover graphic we show another feature of the system wherein a user could aim their iPhone camera at a lamp and then motion with their finger on the display up or down to turn the lamp on or off.

In Apple's patent background they note that a user's surrounding environment often includes multiple electronic devices. For example, a living room may include one or more desktop computers, laptops, keyboards, mice, headphones, cellular telephones, TV, Apple TV box, smart lamps and more. The electronic devices typically communicate with one another over wired connections or over wireless communications links such as Bluetooth and WiFi. communications links.

Users may find it challenging to identify and control which devices are connected and how the devices are connected. Typically, a user must navigate to a settings menu on each individual device, which then provides a list of external devices to which that device is connected.

This process can be cumbersome and unintuitive for the user. There may be multiple devices in the user's environment, making it challenging to manage connections between devices. Apple's latest invention addresses these issues.

According to Apple, an electronic device may include a display system and control circuitry. The display system may be a head-mounted display with the content on the display system being virtual reality content and/or augmented reality content. It could also be an iPhone or iPad display.

The user's environment may be presented on the display system. The environment on the display may be a captured image of the environment or an actual real world viewed through an optical combiner, or could be a completely virtual image representing the environment.

In some scenarios, an electronic device may be used to control external electronic devices. For example, an electronic device may be used to manage connections between external electronic devices and/or may serve as an input-output device for one or more external electronic devices.

The input-output devices may include sensors for identifying external electronic devices within the user's environment, a display system for displaying information about the external electronic devices, and user input components such as touch sensors, force sensors, and other sensors for gathering touch input and/or gesture from a user's fingers.

The electronic device may include communications circuitry for sending signals to and/or receiving signals from the external electronic devices based on the received user input.

For example, a camera may gather images of external electronic devices in the user's surrounding environment, control circuitry may identify and/or gather information about the external electronic devices, a display system may display the camera images in real time, and a user may control or communicate with the external devices by providing touch input to the display or providing other suitable user input.

Computer-generated images (e.g., color-coded lines, virtual buttons, arrows, circles or other shapes, text, menus, colors, etc.) may be overlaid onto the real-time camera images on the display to facilitate user input operations.

The external electronic devices presented on the display system may be captured images of the external electronic devices, may be actual external electronic devices that are viewed through an optical combiner (e.g., a head-mounted display, a head-up display, etc.) or may be virtual, computer-generated images representing the external electronic devices.

Apple's patent FIG. 1 below is a schematic diagram of an illustrative system in which an electronic device with a display system communicates with external electronic devices; FIG. 2 is a perspective view of an illustrative system in which an electronic device presents an environment on a display system and a user provides touch or gesture input to interact with external electronic devices in the environment.

Apple's patent FIG. 3 above is a front view of an illustrative electronic device showing how a finger swipe from a first external electronic device to a second external electronic device may be used to establish a wireless communications link between the first and second external electronic devices in accordance with an embodiment.

Apple's patent FIG. 4 above is a front view of an illustrative electronic device showing how a finger swipe across a computer-generated line extending between first and second external electronic devices may be used to break a wireless communications link between the first and second external electronic devices in accordance with an embodiment.

Apples patent FIG. 5 is a front view of an illustrative electronic device showing how computer-generated display elements that are overlaid onto an environment may have different visual characteristics to indicate different types of wireless communications links in accordance with an embodiment.

Lastly in this first group, FIG. 6 is a front view of an illustrative electronic device showing how touch or gesture input on a computer-generated display element overlaid onto the environment may cause text to appear in the environment.

In group two, we begin with Apple's patent FIG. 7 below which is a diagram of an illustrative system showing how touch or gesture input may be used to control an external electronic device like a living room lamp.  

In patent FIG. 8 below we're able to see is a perspective view of illustrative external electronic devices like MacBooks with features that may be picked up by a camera or other light sensor to identify the external electronic devices in accordance with an embodiment.

Apple's patent FIG. 9 above is a perspective view of an illustrative external electronic device (TV or Display) with visual markers that may be picked up by a camera or other light sensor to identify the external electronic device. Visual markers (#74) in the corners of the device may be ultraviolet ink, visible ink, infrared ink, physical marks, or other suitable visual markers that may be captured by a camera (#20) and read by control circuitry to identify the external devices.

Apple's patent FIG. 10 above is a perspective view of an illustrative external electronic device like a MacBook displaying a bar code that may be picked up by a camera or other light sensor to identify the external electronic device.

And lastly, Apple's patent FIG. 11 above is a perspective view of two displays having a wireless communications link that is established in response to a finger swipe from one of the external electronic devices to the other.

For finer details, review Apple's patent application number 20210067418.

Considering that this is a patent application, the timing of such a product to market is unknown at this time.