The U.S. Patent and Trademark Office officially published a series of 51 newly granted patents for Apple Inc. today. In this particular report we briefly cover three patents covering AR scene camera retargeting and more. Apple was also granted two 'Retail Fixture' design patents, and as always, we wrap up this week's granted patent report with our traditional listing of the remaining granted patents that were issued to Apple today.

Scene Camera Retargeting

HMDs are often used to experience these virtual environments. Various HMDs include a scene camera that captures an image of the environment of the user (e.g., a scene) and a display that displays the image to the user. In some instances, this image or portions thereof can be combined with one or more virtual objects to present the user with an AR experience.

In other instances, the HMD can operate in a pass-through mode in which the image or portions thereof are presented to the user without the addition of virtual objects. Ideally, the image of the real world presented to the user is substantially similar to what the user would see if the HMD were not present.

However, due to the different positions of the eyes, the display, and the camera in space, this may not occur, resulting in impaired distance perception, disorientation, and poor hand-eye coordination. Apple's invention remedies this.

Apple's granted patent covers devices, systems, and methods for scene camera retargeting. In an HMD with a display and a scene camera, the image of the real world presented to the user on the display may not always reflect what the user would see if the HMD were not present due to the different positions of the eyes, the display, and the camera in space. In various circumstances, this results in poor distance perception, disorientation of the user, and poor hand-eye coordination, e.g., while interacting with the real world.

However, in various implementations, compensation for the different positions of the eyes, the display, and the camera is performed by transforming the image captured by the scene camera to make it appear as though it were captured at the same location as the eyes of the user (e.g., to make the captured image appear as though the user were viewing the scene while not wearing the HMD). One form of compensation is based on a planar world assumption: that the real world is an image at a single depth (e.g., a distance from the HMD).

In various implementations, this depth is determined in various ways. Homography on the image performed under this assumption may provide good results with respect to pixels corresponding to objects at this depth.

Apple's patent FIG. 7 below illustrates the image of a scene with certain pixels highlighted.

(Click on image to Enlarge)

In various circumstances, the task is associated with an amount of movement of the user. For example, for a game including a static game board, as in FIG. 7 above, the amount of movement may be small, whereas for an outdoor scavenger hunt game, or Pokemon, the amount of movement may be large.

In various implementations, the device determines the depth in proportion to the amount of movement, where a larger depth is selected when the amount of movement is larger. In various implementations, the device determines the amount of movement by tracking the body of the user or portions thereof using, e.g., accelerometers. In various implementations, the device predicts the amount of movement based on the application being executed by the device.

For those interested in the deeper details, check out granted patent 10,997,741.

Scalability of Multi-Directional Video Streaming."

Developers may also be interested in exploring another Head Mounted Device granted patent 10,999,583 titled "Scalability of multi-directional video streaming." Apple's patent FIG. 2 below figuratively illustrates a rendering application for a sink terminal; FIG. 6 illustrates a method (#600) of exemplary tile download; FIG. 6 illustrates download operations that may occur for a tile that is not being viewed initially but to which the viewport moves during operation.

(Click on image to Enlarge)

Camera with Asymmetric Field of View

Patently Apple has covered a number of Apple patents regarding different aspects of Face ID over the last year, such as these: 01, 02, 03 and 04 and more. Today Apple was granted yet another Face ID patent titled "Camera with asymmetric field of view."

Apple's Patent Abstract: "An imaging device includes objective optics configured to form an image at a focal plane and having an optical axis that intersects the focal plane at an optical center. An image sensor, which includes an array of sensor elements arranged in a matrix of rows and columns, is positioned in the focal plane with a center point of the matrix displaced transversely by at least ten rows relative to the optical center."

Apple's patent FIG. 1 above is a schematic pictorial illustration of an iPhone showing a field of view of its TrueDepth Camera. For more details, review granted patent 10,999,540. 

Key Design Patents Granted Today

The Remaining Patents Granted to Apple Today