The U.S. Patent and Trademark Office officially published a series of 36 newly granted patents for Apple Inc. today. In this particular report we focus on Apple's patent covering a 3D hand tracking system that uses a depth camera like TrueDepth on iPhone X. Beyond Face ID and being able to accommodate Augmented Reality apps, Apple envisions expanding their TrueDepth camera to cover hand gesturing as a new input methodology that could be used for touchless drawing and more.

Another patent of interest today covers Apple's work on Quantum Dot displays. Unfortunately the patent is focused only on LCD and not OLED; though if they ever decide to produce an Apple HDTV, it could still be important. We wrap up this week's granted patent report with our traditional listing of the remaining granted patents that were issued to Apple today.

Apple's leap to introducing their new TrueDepth camera on iPhone X for Face ID could just be the initial application along with AR. Apple could decide to expand this on iDevices and Macs for use with hand gesturing inputs. Back in July Patently Apple posted a report titled "Patents show Apple & Microsoft Racing to bring Hand Gesturing Systems to Computers, Home Appliances & more" It provided an overview of Microsoft's project Prague used on PC's and a Surface Pro tablet.

Apple has delivered on this technology first for Face ID using a dot projector and so Apple's competitors are now bound to try to compete with Apple and expand on it quicker to catch up. But for now, Apple has the lead and in-air gesturing may become a reality thanks in-part to today's granted patent.

Granted Patent: 3D Hand Tracking Using Depth Camera

Apple's newly granted patent covers their invention relating to the field of computer vision and machine learning, and, more particularly, to three-dimensional hand tracking using depth sequences.

As we've never covered this patent as a patent application in the past, as the patent seemed to just land on us a granted patent, we'll provide you with a little patent back ground and summary.

In Apple's patent background they note that "In the field of Human-computer interaction (HCI), i.e., the study of the interfaces between people (i.e., users) and computers, understanding the intentions and desires of how the user wishes to interact with the computer is a very important problem. When handled properly, HCI enables user-friendly interactions, e.g., via multi-modal inputs, such as: voice, touch, body gestures, graphical user interfaces (GUIs), and other input peripherals, such as keyboard, mice, styluses, etc.

The ability to understand human gestures, and, in particular, hand gestures, as they relate to HCI, is a very important aspect in understanding the intentions and desires of the user in a wide variety of applications.

In this invention, a novel system and method for three-dimensional hand tracking is described.

The depth map has the form of an image in which the pixel values indicate topographical information, rather than brightness and/or color of the objects in the scene. Depth maps may equivalently be referred to herein as "3D maps," "depth images," "depth sequences," or "3D images."

Depth maps may be processed in order to segment, identify, and localize objects and their components in the scene. In particular, descriptors (e.g., so-called "features," as will be discussed in further detail below) may be extracted from the depth map based on the depth values of the pixels in a plurality of patches (i.e., areas) distributed in respective positions over objects in the scene that are trying to be identified (e.g., a human hand). Identification of humanoid forms (i.e., 3D shapes whose structure resembles that of parts of a human being) in a depth map, and the exact poses of these parts, which may change from frame to frame, may be used as a means for controlling computer applications.

In Apple's patent FIG. 1A below we're able to see an exemplary sample pixel pairs used to compute depth difference features from a depth map; FIG. 1B illustrates exemplary sample pixel pairs used to compute depth difference features from a depth map, in accordance with some embodiments.

In Apple's patent FIG. 5 noted above we're able to see an exemplary process for performing three dimensional hand tracking using depth sequences.

Apple's invention covers various systems, methods and program storage devices are disclosed, which provide instructions to cause one or more cameras and/or processing units to perform three-dimensional hand tracking using depth sequences.

The techniques disclosed in granted patent are applicable to any number of electronic devices with cameras and displays, such as: digital cameras, digital video cameras, mobile phones, personal data assistants (PDAs), portable music players, monitors, as well as desktop, laptop, and tablet computer displays.

Three dimensional hand tracking has many applications for the Human-computer Interaction (HCI) of the future. Many applications, e.g., touchless drawing, user interaction control, and sign language interaction, may benefit from the introduction of a robust hand tracking application that is able to find and track the three-dimensional location of hands throughout images in a video stream, as well as retain the identification (ID) of particular hands within the scene if there are multiple hands.

More detailed applications of hand trackers, such as pose and gesture detections (e.g., by analyzing the positions of individual fingers on a hand) are also possible, but are not discussed in greater detail in this patent.

Apple's granted patent 9,811,721 was originally filed in Q2 2015 and published today by the US Patent and Trademark Office.

The Remaining Patents granted to Apple Today

Patently Apple presents only a brief summary of granted patents with associated graphics for journalistic news purposes as each Granted Patent is revealed by the U.S. Patent & Trademark Office. Readers are cautioned that the full text of any Granted Patent should be read in its entirety for full details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or behavior will result in being blacklisted on Disqus.