Today the US Patent & Trademark Office published a patent application from Apple that relates to Apple's future HMD, and more specifically to displays, cameras and image sensors. The system is designed to use foveation techniques that ensures that the highest quality images are at the center of the user's vision. The periphery of a user's vision will deliver a lower quality image. The user won't even notice the images are split in this way and it will reduce power demand on the HMD.

Apple notes in their patent filing that when high-resolution images are being displayed for a viewer, it may be burdensome to display images at full resolution across an entire display. Foveation techniques involve displaying only critical portions of an image at full resolution and can help reduce the burdens on a display system.

In some cases, images of the environment of the user may be displayed on the display. However, it may be burdensome to use the image sensor to obtain high-resolution images of the user's entire environment.

Apple's invention covers a display for their Head Mounted Device (HMD) that may be a transparent display so that a user may observe real-world objects through the display while computer-generated, or Augmented Reality content is overlaid on top of the real-world objects by presenting computer-generated images on the display.

The display may also be an opaque display that blocks light from real-world objects when a user operates the HMD. In this type of arrangement, a pass-through camera may be used to display real-world objects to the user.

The pass-through camera may capture images of the real world and the real-world images may be displayed on the display for viewing by the user. Additional computer-generated content (e.g., text, game-content, other visual content, etc.) may optionally be overlaid over the real-world images to provide an augmented reality environment for the user.

The display may be a foveated display. Using a gaze-tracking system in the head-mounted device, the device may determine which portion of the display is being viewed directly by a user.

A user will be less sensitive to artifacts and low resolution in portions of the display that lie within the user's peripheral vision than portions of the display that are being directly viewed. Accordingly, the device may display different portions of an image with different resolutions, as noted in patent FIG. 3 further below.

The pass-through camera may capture some high-resolution image data for displaying on the display. However, only low-resolution image data may be needed to display low-resolution images in the periphery of the user's field of view on the display. Therefore, the pass-through camera may only capture high-resolution images that correspond to the portion of the user's field-of-view that is being directly viewed and may capture lower resolution image data that corresponds to the real-world objects in the user's peripheral vision.

Adjusting the pass-through camera to only capture high-resolution image data in selected portions of the user's field of view may reduce processing burden and power consumption within the head-mounted device.

Apple's patent FIG. 1 below is a schematic diagram of an overview of an illustrative head-mounted device illustrating the component makeup of the HMD; FIG. 2 is a top view of an illustrative head-mounted device; FIG. 3 is a diagram showing how high-resolution images may be displayed in a first portion of a user's field of view whereas low-resolution images may be displayed in a second portion of a user's field of view.

Apple's patent FIG. 16 above is a cross-sectional side view of an illustrative camera module that includes a first image sensor for capturing high-resolution images, a second image sensor for capturing low-resolution images, and a beam-splitter.

More specifically, the control circuitry in the head-mounted device (e.g., control circuitry #50 in FIG. 1) may dynamically select which portions of the high-resolution image sensor #102H and/or low-resolution image sensor #102L to read out. The image data may then be combined to form a single image with high-resolution image data in desired portions and low-resolution image data in the remaining portions.

Apple's patent application was filed in late March 2020 and published today by the U.S. Patent Office. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Apple inventors

Mr. Gross: Image Scientist Manager of Image Science Team

Mr. Schmuck: Senior Engineering Program Manager, Product Design

Mr. Sauers: Mechanical Product Designer

Mr. Huo: Product Design Engineer