The U.S. Patent and Trademark Office officially published a series of 41 newly granted patents for Apple Inc. today. In this particular granted patent report we cover one of Apple's acquired patents from PrimeSense. The patent covers a Kinect-like device that creates 3D depth maps of a human hand. The mapping would allow a user to control future applications with hand signals that could apply to a computer system and technically to a device like Apple TV. A number of other PrimeSense patents have recently come to light. One covered a "3D Mapping Laser Beam Projector System," another covered a 3D Scanning Engine and yet another related to enhanced face detection using depth-information. How Apple intends to use PrimeSense's advanced 3D technologies in the future is unknown at this time.

Granted Patent: 3D Depth Maps of a Scene containing a Human Hand

Apple's newly granted patent covers their invention relating to methods and systems for three-dimensional (3D) mapping, and specifically to extracting high-level information from depth-map data.

Patent Background

A number of different methods and systems are known in the art for creating depth maps. In the present patent application and in the claims, the term "depth map" refers to a representation of a scene as a two-dimensional matrix of pixels, in which each pixel corresponds to a respective location in scene and has a respective pixel depth value, indicative of the distance from a certain reference location to the respective scene location. (In other words, the depth map has the form of an image in which the pixel values indicate topographical information, rather than brightness and/or color of the objects in the scene.) Depth maps may equivalently be referred to as 3D maps, depth images, or 3D images.

Depth maps may be processed in order to segment, identify and localize objects and their components in the scene. Identification of humanoid forms (meaning 3D shapes whose structure resembles that of a human being) in a depth map, and the exact poses of these forms, which may change from frame to frame, may be used as a means for controlling computer applications.

Basic Overview of Apple's Granted Patent

An embodiment of the present invention provides a method for processing data, including: receiving a depth map of a scene containing a human hand, the depth map including a matrix of pixels having respective pixel depth values; extracting from the depth map respective descriptors based on the depth values in a plurality of patches distributed in respective positions over the human hand; matching the extracted descriptors to previously-stored descriptors in a database; and estimating a pose of the human hand based on stored information associated with the matched descriptors.

Typically, estimating the pose includes applying kinematics based on anatomical constraints of the hand in processing the descriptors.

In a disclosed embodiment the method includes receiving a color or grayscale image of the human hand, and extracting the descriptors includes incorporating information from the color or grayscale image in the descriptors together with the depth values.

In a further disclosed embodiment estimating the pose includes detecting that a part of the hand is occluded in the depth map, and excluding the occluded part from estimation of the pose. Typically, estimating the pose includes choosing a most anatomically probable hand configuration in response to detecting that the part of the hand is occluded.

In a yet further disclosed embodiment estimating the pose includes expressing the pose in terms of a hand posture description language. The hand description language may include assigning one of a set of positions to each finger of the hand so as to define postures of the hand. Defining postures of the hand may include defining one or more invalid postures of the hand, and estimating the pose of the hand may include excluding the one or more invalid postures.

In an alternative embodiment the previously-stored descriptors in the database are associated with corresponding pointers to respective locations of anatomical landmarks on the human hand, and estimating the pose includes applying the pointers to the respective positions of the patches from which the matching descriptors were extracted in order to estimate the locations of the landmarks.

Typically, the landmarks include at least one element selected from a group including a fingertip, a joint, a palm plane, and a base of the hand. Estimating the pose may include reconstructing the pose by applying reverse kinematics using at least one of the locations of the landmarks. The method may include creating the database by processing a set of training maps in which ground-truth locations of the anatomical landmarks are indicated in order to find the corresponding pointers. Alternatively or additionally, estimating the pose may include associating respective weights with the estimated locations of the landmarks, and applying a weighted voting process using the weights to find the locations of the landmarks.

Apple's patent FIG. 1 is a schematic, pictorial illustration of a system for 3D mapping and tracking of a human hand; FIG. 2 is a schematic representation of a depth map of a hand; FIG. 3 is a schematic image showing an estimated hand skeleton superimposed on the hand in the depth map of FIG. 2.

Apple's patent FIG. 4A is a schematic image showing descriptor centers and a descriptor patch used in estimating the pose of the hand of FIG. 2; FIG. 4B is a schematic image showing a point cloud corresponding to the estimated location of the joint of the index finger of the hand of FIG. 2.

In patent FIG. 4B noted above, the cloud corresponds to estimated locations #90 of the index fingertip joint, and a circle #96 indicates the location with highest density, which is taken to be the fingertip joint location. In addition, from skeleton #60 a location #98 of the index fingertip itself may be estimated.

Apple credits Israeli engineers Shai Litvak, Leonid Brailovsky and Tomer Yanir as the inventors of this granted patent which was originally filed in Q1 2013 – although the work associated with this invention dates back to 2011.

Patently Apple presents only a brief summary of granted patents with associated graphics for journalistic news purposes as each Granted Patent is revealed by the U.S. Patent & Trademark Office. Readers are cautioned that the full text of any Granted Patent should be read in its entirety for full details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Comments are reviewed daily from 5am to 7pm MST and sporadically over the weekend.