Earlier today Patently Apple posted an IP report titled "For Sports Fans and Developers, Apple has filed a patent for Posture and Motion Monitoring using Mobile Devices." Apple's second patent on this subject published by the U.S. Patent Office on Thursday, is titled "Full Body Pose Estimation through Feature Extraction from Multiple Wearable Devices."

Apple notes in their patent background that wearable devices are now prevalent in society. Many consumers wear both smartwatches and ear devices on a daily basis. Many of these wearable devices include inertial sensors that can sense the motion of the user. For example, inertial sensors embedded in a smartwatch can capture a user's arm motion which can then be used by a fitness application to compute various fitness metrics, such as the amount of calories burned by the user during exercise. Some ear devices (e.g., earbuds) include inertial sensors that control audio playback, provide microphone orientation data used in beamforming to reduce background noise, and head tracking for anchoring a spatial audio sound field played through the ear devices.

Apple's invention  covers full body pose estimation through feature extraction from multiple wearable devices.

In an embodiment, a method comprises: obtaining, with at least one processor, point of view (POV) video data and inertial sensor data from multiple wearable devices worn at the same time by a user; obtaining, with the at least one processor, depth data capturing the user's full body; extracting, with the at least one processor, two-dimensional (2 D) keypoints from the POV video data; reconstructing, with the at least one processor, a full body 2 D skeletal model from the 2 D keypoints; generating, with the at least one processor, a three-dimensional (3 D) mesh model of the user's full body based on the depth data; merging, with the at least one processor, nodes of the 3 D mesh model with the inertial sensor data; aligning, with the at least one processor, respective orientations of the 2 D skeletal model and the 3 D mesh model in a common reference frame; and predicting, using a machine learning model, classification types based on the aligned 2 D skeletal model and 3 D mesh model.

In an embodiment, at least one wearable device is a headset worn on or in the ears of the user and at least one wearable device is smartwatch worn on a wrist of the user.

In an embodiment, the method is performed by a central computing device that is wireless coupled to the wearable devices.

In an embodiment, altimeter data obtained from the multiple wearable devices is used to align respective orientations of the 2 D skeletal model and the 3 D mesh model in the common reference frame.

Other embodiments can include an apparatus, computing device and non-transitory, computer-readable storage medium.

Particular embodiments disclosed provide one or more of the following advantages. The disclosed embodiments provide a more accurate full body skeletal model that can provide detailed insights into a user's movements, allowing improved progress tracking for various fitness activities or health monitoring.

For example, a user can be provided a workout summary on their smartphone or smartwatch that includes repetition data for various exercises (e.g., number of weighted squats or pushups), and trend data such as range of motion improvements.

Apple's patent FIG. 1 below is a block diagram of a system for full body pose estimation through features extracted from multiple wearable devices; FIG. 2 shows various locations of wearable devices for various types of workouts.

Apple's patent FIG. 3A below is a block diagram of a system for full body pose estimation using features extracted from multiple wearable devices;  FIG. 3B is flow diagram of a training process for a machine learning (ML) model for full body pose estimation using features extracted from multiple wearable devices.

Apple's patent FIG. 4 above is a flow diagram of a process of full body pose estimation using features extracted from multiple wearable devices.

For finer details, review Apple's patent application number US 20230101617 A1.

Apple Inventors

• Vicki Powell: Health Tech Hardware Engineer
• Wes Zuber; Sensing HW Incubation Engineer   
• Miki Olivia Hansen: No LinkedIn Profile found