The U.S. Patent Office has Published Facebook's Patent behind their Future AR Glasses Wristband Input Device
In March Patently Apple posted a report titled "Facebook reveals initial work on wrist-based input devices for future AR Glasses & plans to reveal their work on Haptic Gloves later this year. An image of the wrist device from their research department is presented above. You could some of the details behind this project from our March report here. One of Facebook's promotional videos on this future device is presented below.
Over the weekend, I discovered Facebook's July 15th patent application supporting their wrist device titled "Wearable Electronic Devices and Extended Reality Systems including Neuromuscular Sensors."
The invention's technology relates to systems and methods that detect and interpret neuromuscular signals for use in performing actions in an augmented reality (AR) environment as well as other types of extended reality (XR) environments, such as a virtual reality (VR) environment, a mixed reality (MR) environment, and the like.
Wearable EMG Sensor Bands
Facebook's patent FIG. 8A below illustrates a wearable system with a plurality of sensors (#810) arranged circumferentially around an elastic band (#820) structured to be worn around a user's lower arm or wrist. The sensors may be neuromuscular sensors (e.g., EMG sensors). As shown, there may be sixteen sensors arranged circumferentially around the elastic band at a regular spacing.
A wearable system (e.g., armband, wristband, thighband, etc.) can be used to generate control information for controlling a robot, controlling a vehicle, scrolling through text, controlling a virtual avatar, and/or performing any other suitable control task.
In some embodiments, the sensors may include only a set of neuromuscular sensors (e.g., EMG sensors). In other embodiments, the sensors may include a set of neuromuscular sensors and at least one auxiliary device. The auxiliary device(s) may be configured to continuously sense and record one or a plurality of auxiliary signal(s).
Examples of auxiliary devices include, but are not limited to, IMUs, microphones, imaging devices (e.g., cameras), radiation-based sensors for use with a radiation-generation device (e.g., a laser-scanning device), heart-rate monitors, and other types of devices, which may capture a user's condition or other characteristics of the user.
As shown in FIG. 8A below, the sensors may be coupled together using flexible electronics (#830) incorporated into the wearable system. FIG. 8B illustrates a cross-sectional view through one of the sensors of the wearable system shown in FIG. 8A.
(Click on image to enlarge)
Facebook's patent FIG. 10 below illustrates a diagram schematically showing an example of an implementation using EMG sensors and a camera. For example, FIG. 10 shows a user's arm and an attached hand ("arm/hand" #1010), which is made up of one or more joints and segments, and which can be depicted as a musculoskeletal representation.
More particularly, the user's hand segments (#1020) are connected by joints. The arm and hand positions and segment lengths of the arm and the hand can be determined by the system 1000 and positioned within a three-dimensional space of a model musculoskeletal representation. Further, the user's hand may also include an interpolated forearm segment (#1030).
(Click on image to Enlarge)
Facebook's engineers further note that the neuromuscular signals obtained by the plurality of neuromuscular sensors and the image(s) captured by the camera(s) may be used, for example, to determine a force that that the user is applying to the physical object. Neuromuscular signal data and auxiliary sensor data (e.g., camera data) may be combined in any other suitable way to determine information associated with the user's interaction with the physical object, and embodiments are not limited in this respect.
The process then proceeds to act where feedback based on the determined information about the interaction of the user with the physical object is provided. In some embodiments, the feedback is provided to the user interacting with the object. For example, the AR-based system may provide feedback (e.g., visual feedback, auditory feedback, haptic feedback) to the user within the AR environment.
In embodiments where visual feedback is provided within the AR environment, the visual feedback may be provided in any suitable way. For example, the physical object with which the user is interacting may change colors or glow indicating that the user is interacting with the object.
Alternatively, the feedback may be provided using a visual indicator separate from the physical object. For example, an icon or other visual indicator may be displayed within the AR environment showing an interaction mode (e.g., paintbrush mode) for the object with which the user is interacting.
In some embodiments that provide feedback to the user, the feedback may be provided using non-visual forms of feedback such as auditory or haptic feedback. The feedback may, for example, instruct the user that the physical object that he/she is interacting with may have augmented properties or functions that may not be available through ordinary real-world interactions with the object.
Wearable EMG Sensor Patches
Another aspect of Facebook's patent covers a toned down application for EMG sensors in the form of wearable patches. In some embodiments, the one or more sensor(s) (#110) may comprise a plurality of sensors and at least some of the plurality of sensors may be arranged as a portion of a wearable device structured to be worn on or around a part of a user's body.
Alternatively, at least some of the sensors may be arranged on a wearable patch structured to be affixed to a portion of the user's body. FIGS. 6A-6D show various types of wearable patches.
Facebook's patent FIG. 6A below shows a wearable patch (#62) in which circuitry for an electronic sensor may be printed on a flexible substrate that is structured to adhere to an arm, e.g., near a vein to sense blood flow in the user. The wearable patch may be an RFID-type patch, which may transmit sensed information wirelessly upon interrogation by an external device.
Facebook's patent FIG. 6B below shows a wearable patch (#64) in which an electronic sensor may be incorporated on a substrate that is structured to be worn on the user's forehead, e.g., to measure moisture from perspiration. The wearable patch (#64) may include circuitry for wireless communication, or may include a connector structured to be connectable to a cable, e.g., a cable attached to a helmet, a heads-mounted display, or another external device.
The wearable patch may be structured to adhere to the user's forehead or to be held against the user's forehead by, e.g., a headband, skullcap, or the like.
Facebook's patent FIG. 6C above shows a wearable patch 66 in which circuitry for an electronic sensor may be printed on a substrate that is structured to adhere to the user's neck, e.g., near the user's carotid artery to sense flood flow to the user's brain. The wearable patch 66 may be an RFID-type patch or may include a connector structured to connect to external electronics. FIG. 6D shows a wearable patch 68 in which an electronic sensor may be incorporated on a substrate that is structured to be worn near the user's heart, e.g., to measure the user's heartrate or to measure blood flow to/from the user's heart. As will be appreciated, wireless communication is not limited to RFID technology, and other communication technologies may be employed. Also, as will be appreciated, the sensors 110 may be incorporated on other types of wearable patches that may be structured differently from those shown in FIGS. 6A-6D.
For more on this, review Facebook's patent application 20210217246.
Patently Apple covered Motorola's Smart Skin Tattoo patent. Technology Dreamer Regina E. Dugan, who served as the 19th Director of the Defense Advanced Research Projects Agency (DARPA), also worked at Google and Facebook. So the wearable patches that the Facebook patent currently describes has a history going back to Dugan's original work at Facebook.