On Thursday, the US Patent & Trademark Office published a patent application from Apple that relates to a new multi-gesture-based Apple Watch control system that could control devices such lamps, Apple TV, HomePod and other at a distance.

Apple noted in their patent background that electronic devices such as wearable electronic devices are often provided with input components such as keyboards, touchpads, touchscreens, or buttons that enable a user to interact with the electronic device. In some cases, an electronic device can be configured to accept a gesture input from a user for controlling the electronic device.

 Aspects of Apple's invention provides for gesture control, using a finite set of gestures detectable at a first electronic device, of various different control elements of various different second electronic devices, such as internet-of-things (IoT) devices and/or other types of electronic devices.

For example, a home environment may include multiple devices that each has a set of one or more control elements, such as switches, knobs, dials, buttons, and the like. As examples, the multiple devices may include an IoT device (lamps, thermostat, Apple TV etc.)and a smart speaker/HomePod. The IoT device may be a network-connected light source and may include a set of control elements such as an on-off switch and a dimmer. The smart speaker may include a different set of control elements such as a play button, a stop button, a pause button, and a volume knob.

In accordance with one or more implementations, a first electronic device such as an Apple Watch can provide for (i) gesture-based selection (e.g., using a first sensor such as an ultra-wide band (UWB) sensor, a Bluetooth Low Energy (BLE) sensor, an image sensor coupled with a computer-vision engine, etc.) of another device (e.g., from among multiple devices) for gesture control, (ii) surfacing of an appropriate set of control element icons for the selected device (e.g., as stored at the first electronic device during a registration or pairing of the selected device for gesture control), (iii) gesture-based selection (e.g., using one or more other sensors such as inertial measurement unit (IMU) sensors including one or more of an accelerometer, a gyroscope and a magnetometer) of one of the set of control element icons for the selected device, and (iv) gesture-based control (e.g., using one or more additional sensors such as electromyography (EMG) sensors) of the selected one of the set of control element icons.

Apple's patent FIG. 3 below illustrates a perspective view of an example electronic device, an Apple Watch which is the focus controlling device; FIG. 4 illustrates a use case in which an Apple Watch is in proximity of multiple other devices that are available for gesture control; FIG. 5 illustrates a use case in which an Apple Watch tracks the locations of multiple other devices that are available for gesture control.

Apple's patent FIG. 7 below illustrates a use case in which an Apple Watch brings forward a gesture-control icon for a gesture-control element of the selected device of FIG. 6 (above); FIG. 8 illustrates a use case in which an Apple Watch  receives a selection of different other devices for gesture control; FIG. 9 illustrates a use case in which an Apple Watch brings forward a different set of gesture-control icons for gesture control of the selected device of FIG. 8.

Apple's patent FIG. 11 below illustrates a use case in which an Apple Watch detects a gesture for operation of a selected gesture-control icon; FIG. 12 illustrates a use case in which an Apple Watch receives a selection of another gesture-control icon; FIG. 13 illustrates a use case in which an electronic device detects another gesture for operation of another selected gesture-control icon.

Further to patent FIG. 11, once the gesture-control icon #902 has been selected (e.g., as indicated by the indicator 1000), the Apple Watch #107 may detect a gesture #1102 for interaction with the selected gesture-control icon #902. In this example, the gesture may be a finger movement gesture that can be detected by the watch as a button push gesture for pressing the play-pause button represented by the gesture-control icon.

In this example, the representation of the gesture-control icon as a button may help to inform the user that a button push gesture can be used to control the smart speaker/HomePod and thereby elicit the desired control gesture from the user (e.g., without having to provide an explanation to the user in words and/or text as to how to operate the HomePod using the watch.

The gesture may be recognized by receiving sensor data from a sensor (e.g., an image sensor, an EMG sensor, etc.) of the Apple Watch and provide the received sensor data to a machine learning model trained to recognize a predetermined set gestures including a button-push gesture based on input sensor data from at least the sensor.

Apple's patent FIG. 14 below illustrates a flow chart of an example process for operating an electronic device for gesture control of another device in accordance with one or more implementations.

(Click on Flow Chart below to Enlarge)

For more details, review Apple's patent application # US 20230076716 A1.

Apple Inventors

• Joseph Cheng: Platform Architect
• Kaan Dogrusoz: ML & Sensing R&D
• Ali Moin: AI / ML Researcher
• Erdrin Azemi: Manager AI/ML Research

Google Patent, Similar Idea but for Smartglasses

For interest sake, on February 16, the U.S. Patent Office published a patent from Google with a similar idea, though aimed at controlling a smartglasses User Interface as noted in patent FIGS. 2 and 3 below.

(Click on patent figures below to Enlarge)

The patent abstract for Google's patent #20230045333 is as follows:

"A smartwatch is configured to be usable to control a peripheral device. The smartwatch can be toggled between a typical or usual operating mode for a smartwatch and a mode for controlling the peripheral device. The toggling function is bound to a virtual button. Force applied to the virtual button is measured so that multiple functions, including the toggling function, can be accessed by applying different amounts of force to the button for a predetermined amount of time."