We all know by now that Apple protected Vision Pro with over 5,000 patents and the U.S. Patent Office is publishing dozens of patents almost on a weekly basis. This report lightly zero's in on two patents related to hand matting and synthesized hand textures when using Vision Pro now or in the future.

Hands Matting Responsive to Low Light Conditions

Apple's invention relates to managing hands matting based on light conditions includes receiving image data comprising a hand, and determining a brightness in the image data. If the brightness in the first image data satisfies the brightness threshold, hands matting is performed on the image data to extract the hand from the image data and the image data is blended with virtual content in accordance with the extracted hand. If the brightness in the first image data satisfies a disablement threshold, hands matting is disabled and the image data is blended with virtual content without extracting the hand. An alternative representation of the location of the hand is depicted in the final image.

A user-specific hand mesh and/or user-specific hand texture may be obtained. According to one or more embodiments, the user-specific hand mesh and/or the user-specific hand texture may be determined, in part, based on enrollment data. For example, an enrollment process may include prompting enrollment of a user's hands such that the system can capture representative views of the user's hand or hands, from which geometry and texture specific to the user can be determined. That texture and geometry can then be retried and used to determine a current user-specific mesh and texture. The flowchart then optionally proceeds to block 765, where the virtual representation of the hand is rendered using the user-specific hand mesh and hand texture. In accordance with the hand tracking data.

Apple's patent FIG. 8 below shows an example diagram of a view of a composite image including a rendered hand mesh and virtual content; FIG. 9 shows an example diagram of a view of a composite image including a virtual representation of a hand and virtual content.

Apple's patent FIG. 7B above depicts an example flow diagram for using a hand mesh to blend image data captured by a camera with virtual content. For full details, review patent application 20240404186

Synthesized Hand Texture

This second Apple invention relates to various implementations that includes devices, systems, and methods that provide a matching skin tone between the hands and the face of a three-dimensional (3D) user representation.

For example, a process may obtain at least two skin tones corresponding to colors representing at least two different portions of a hand of a user. In response, a hand texture representing a spatial arrangement of color representing an appearance of the hand is determined. The hand texture may be determined based on the at least two skin tones and hand texture sample data. The process may further include determining a hand structure associated with physical attributes of the hand and the hand structure and the hand texture may be provided for use in generating a 3D representation of the hand.

Apple's patent FIG. 2 below illustrates an example system that uses a skin tone of a palm side and a skin tone of a back side of a hand of a user in combination with delighted facial texture and sample hand textures to generate a synthesized hand texture; FIG. 3 illustrates an example process using a neural network to estimate a nominal skin tone of a hand(s) of a user during an enrollment process for generating a user representation.

Apple's patent FIG. 4 above is a flowchart illustrating an exemplary method that determines a hand texture for use in providing a 3D representation of a user's hand and/or body

For full details, review Apple's patent application 20240404171

Ten Other HMD Related Patents Published Last Thursday

With Apple filing 5,000+ patents just for Vision Pro and continuing to file patent for future features for both full headsets and smartglasses, it's impossible to cover each individually. Below are an additional ten patents from Apple that engineers and those interested in Apple's HMD projects can review at their own leisure.

• 20240406362 Adaptive Virtual Content Magnification
• 20240406364 Recording Content in a Head-Mounted Device
• 20240398053 Adjustable Headbands
• 20240406334 Color Correction For Electronic Device With Immersive Viewing
• 20240402869 Devices, Methods and GUIs for Content Collaboration and Sharing
• 20240404210 GUIs for Generating Reminders for a User Experience Session in and Extended Reality Environment
• 20240402800 Input Recognition In 3D Environments
• 20240402394 Lens Coatings
• 20240402821 input Recognition Based On Distinguishing Direct And Indirect User Interactions
• 20240404213 Composing of Images from Virtual and Physical Cameras