Last Thursday the US Patent & Trademark Office published a patent application from Apple that relates to monitoring the posture and motion of a user while the user is engaged in a physical activity, such as exercise or a sport like golf. Diehard sports and athletic types will appreciate such a new feature on future Apple devices.

Apple notes that many consumers own a smartphone, a smartwatch and a headset (e.g., ear buds). These devices all include motion sensors, such as accelerometers, gyroscopes and altimeters. These devices typically include one or more video cameras. Some of the more advanced devices also include depth sensors, such as a TOF sensor. The camera data and/or depth data can be used to generate a skeletal model of human body.

Techniques for generating two-dimensional (2D) skeletal models from camera data are well-known. Techniques for lifting the 2D skeletal models to three-dimensional (3D) skeletal models are also well-known. A 3D skeletal model provides an excellent resource for determining the full body posture of a user.

The skeletal model typically includes joint positions for all of the joints of the human body, such as ankles, knees, hips, waist, shoulders, elbows, wrists and neck.

In some embodiment, the skeletal model can be used to generate a skeletal mesh. The positions are typical in a camera and/or depth sensor reference frame, which is often different than the reference frames of inertial sensors. Additionally, the data rates for motion data and skeletal data may be different depending on the device and/or application.

Apple further notes that it is desirable to use motion data and skeletal data together to estimate a user's body pose. This information can be used in a variety of applications, such as monitoring golf swings, monitoring Yoga poses, gaming applications (e.g., TV with 'camera on body motion' sensors), people disambiguation in a multi-person scene based on the uniqueness of per-user behavior captured by the respective motion sensors and remote monitoring by fitness trainers or coaches.

Apple's patent FIGS. 1A-1D below illustrate an application of posture and motion monitoring for monitoring a user's golf swing.

More specifically, FIGS. 1A and 1B, the estimated body pose can be used to determine if the user's golf form or stance is correct. This can be done by kinematic modeling of a position and orientation of a smartwatch worn on the wrist of the user. For example, the motion data provided by the smartwatch can be used to measure the physical forces at impact during the swing. Additionally, the skeletal data can be used to detect the user's form and stance by monitoring the position of the elbow and shoulder joints of the user, for example.

(Click on patent figure below to Enlarge)

FIGS. 1A-1D above illustrate one example application that can use the estimated body pose generated from motion data provided by different mobile devices located on different parts of the user's body in combination with 2D or 3D skeletal data and/or a skeletal mesh. The fusion of motion data and skeletal data will is described in patent FIG. 2 below.

(Click on patent figure below to Enlarge)

Apple's patent FIG. 2 above is a block diagram of system #200 for monitoring posture and motion with mobile devices. The system includes an Apple Watch #201, skeletal data generator #202, calibrator #203, communication manager #204, synchronizer #205 and body pose estimator #206 which further comprises an interpolator #210 and estimation filter #211 (e.g., a Kalman filter). In an embodiment, the system can also be implemented on an iPhone, iPad, a network computer and other devices.

Apple's patent FIGS. 3 and 4 below illustrate operation of the calibrator shown in FIG. 2.

Apple's patent FIG. 5 above illustrates predicting joint positions using motion data and skeletal data.

For finer details, review Apple's patent application number US 20230096949 A1. For sports fans or developers interested in this subject matter, stay tuned for our secondary patent report on this subject matter later today.