The U.S. Patent and Trademark Office officially published a series of 50 newly granted patents for Apple Inc. today. In this particular report we briefly cover two granted patents/inventions that derived from PrimeSense, an Israeli company Apple acquired that has already delivered Face Id for Apple. The patents cover future use of in-air control gesturing that could be used with future Macs, iDevices and Apple TV. Since the work on this has been ongoing since at least 2009, the patent figures are outdated, though the technology is not.  

Enhanced 3D Interfacing: In-Air Gesturing

Apple's first granted patent that we're covering is 10,719,214. The patent covers an invention that provides a method for operating a computerized system, which is carried out by presenting user interface elements on a display screen of the computerized system and detecting a first gesture made in a three-dimensional space by a distal portion of an upper extremity of a user while a segment of the distal portion thereof rests on a surface. In response to the first gesture, an area of the display screen selected by the user is identified, and a corresponding user interface element is displayed. After displaying the corresponding user interface element, a second gesture made by the distal portion while the segment continues to rest on the surface is detected so as to select one of the user interface elements that appears in the selected area.

In another embodiment, the method further includes mapping an operation to the corresponding user interface element, wherein the second gesture causes the operation to be performed.

Apple's patent FIG. 3, which is presented above as our cover graphic, is a view of portions of a system operating under remote control of a user. In the example of FIG. 3, the user performs a leftward, substantially horizontal gesture using one hand that is reflected on a computer display as indicated by the arrow #52.

This has been an ongoing project at Apple and over the years we've covered many aspects of the invention that will one day deliver in-air gesturing to control various features on Macs, iDevices and even Apple TV as a modern application for this invention. A few of the other patents on this technology could be reviewed in these other patent reports: 01 (the original patent granted today), 02, 03, 04 and 05.

Scanning Projectors & Image Capture modules for 3D Mapping

With the advancement of Face ID technology, Apple's granted patent 10,721,459 covers methods and apparatus for efficient projection of patterns, particularly for 3D mapping, as well as for imaging of such projected patterns. This is another project that stemmed from Apple's PrimeSense team stationed in Israel.

Apple's patent FIG. 1 noted above is a schematic top view of a system for 3D mapping. The depth information in the 3D maps generated by device 22 may be used, for example, by a host computer (not shown) as part of a 3D user interface, which enables the user to interact with games and other applications running on the computer and with elements shown on a display screen.

Apple's patent FIG. 6 below is a more in-depth schematic of pictorial view of a 3D mapping system #90 in operation. In this system, device #22 is used in conjunction with a game console #92 to operate an interactive game.

For this purpose, device #22 projects a pattern of spots #100 onto objects in its field of view, including the participants and a background #98, such as the wall (and other elements) of the room in which system #90 is located. The device captures and processes an image of the pattern in order to create a 3D map of the participants and other objects in the scene. The gaming console controls the game in response to the participants' body movements, which are detected by device #22 or console #92 by segmenting and analyzing changes in the 3D map.

This PrimeSense technology was first used in the original Xbox with Kinect device. Apple yanked the technology and Microsoft was forced to find an alternative to replace it with.

In today's granted patent Apple is further tweaking their patent claims. Below are just three of changes that we discovered.

Claim #8: The apparatus according to claim 7, wherein the processor is configured to modulate the visible light beam so as to project the visible image onto the object responsively to the 3D map.

Claim #9: The apparatus according to claim 8, wherein the visible image is tailored to a shape of the object.

Claim #13: The method according to claim 12, and comprising identifying a location of the object in the region of interest, and adjusting a scan range of the scanner based on the identified location so as to map the object in the 3D map.

A few of the other Apple patents covering this project include: 01 (2015), 02 (2015), 03 (2016) and 04 (March 2020).