Today the US Patent & Trademark Office published a patent application from Apple that relates to a mixed-reality headset supporting a high-quality Foveated display. Apple's original work on Foveated displays was originally filed back in 2017, published and reported on in June 2019. In June, Patently Apple covered another patent on this in a report titled "Apple's 8K Foveated Display Technology could apply to both Mini-Displays for a VR Headset & a Wall Mounted Display+." Apple was granted their first patent related to Foveated displays earlier this month.

In today's patent application, Apple notes that in general, their invention relates to head-mounted devices, and, more particularly, to head-mounted devices with displays and image sensors.

Electronic devices often include displays and image sensors. Particularly when high-resolution images are being displayed for a viewer, it may be burdensome to display images at full resolution across an entire display. Foveation techniques involve displaying only critical portions of an image at full resolution and can help reduce the burdens on a display system. In some cases, images of the environment of the user may be displayed on the display. However, it may be burdensome to use the image sensor to obtain high-resolution images of the user's entire environment.

Mixed Reality Head-Mounted Device with a High Quality Foveated Display

Apple's invention covers an electronic device such as a head-mounted device may have a display. In some cases, the display may be a transparent display so that a user may observe real-world objects through the display while computer-generated content is overlaid on top of the real-world objects by presenting computer-generated images on the display. This is what makes is a Mixed Reality headset supporting both AR and VR applications.

The display may also be an opaque display that blocks light from real-world objects when a user operates the head-mounted device. In this type of arrangement, a pass-through camera may be used to display real-world objects to the user.

The pass-through camera may capture images of the real world and the real-world images may be displayed on the display for viewing by the user. Additional computer-generated content (e.g., text, game-content, other visual content, etc.) may optionally be overlaid over the real-world images to provide an augmented reality environment for the user.

The display may be a foveated display. Using a gaze-tracking system in the head-mounted device, the device may determine which portion of the display is being viewed directly by a user. A user will be less sensitive to artifacts and low resolution in portions of the display that lie within the user's peripheral vision than portions of the display that are being directly viewed. Accordingly, the device may display different portions of an image with different resolutions.

The pass-through camera may capture some high-resolution image data for displaying on the display. However, only low-resolution image data may be needed to display low-resolution images in the periphery of the user's field of view on the display. Therefore, the pass-through camera may only capture high-resolution images that correspond to the portion of the user's field-of-view that is being directly viewed and may capture lower resolution image data that corresponds to the real-world objects in the user's peripheral vision.

Adjusting the pass-through camera to only capture high-resolution image data in selected portions of the user's field of view may reduce processing burden and power consumption within the head-mounted device.

There are a number of possible arrangements for the pass-through camera that allow the camera module to selectively capture high-resolution images. For example, the front-facing camera may include an image sensor with two or more pixel densities, a distortion lens, and/or one or more planar or curved mirrors. Any of the components in the camera module may be adjusted to change which portion of a scene is captured with high-resolution image data.

Apple's patent FIG. 1 below is a schematic diagram of an illustrative head-mounted device; FIG. 2 is a top view of an illustrative head-mounted device; and FIG. 3 is a diagram showing how high-resolution images may be displayed in a first portion of a user's field of view whereas low-resolution images may be displayed in a second portion of a user's field of view.

Apple's patent FIG. 4 below is a cross-sectional side view of an illustrative camera module that includes an image sensor with a varying pixel density that is positioned by positioning equipment; FIG. 6 is a cross-sectional side view of an illustrative camera module that includes a distortion lens that is positioned by positioning equipment in accordance with an embodiment; FIG. 7 is a cross-sectional side view of an illustrative camera module that includes a curved mirror that is positioned by positioning equipment

Apple's patent FIG. 16 below s a cross-sectional side view of an illustrative camera module that includes a first image sensor for capturing high-resolution images, a second image sensor for capturing low-resolution images, and a beam-splitter; FIG. 17 is a cross-sectional side view of an illustrative camera module that includes an image sensor and a lens that may change shape to control how light is directed to the image sensor.

Apple's patent FIG. 18 above is a flow chart of illustrative operations involved in operating a head-mounted device with a gaze-tracking system and a front-facing camera.

According to Apple, the head-mounted device may optionally include adjustable components stacked in series with display. For example, the head-mounted device may include an adjustable polarizer (e.g., a polarizer with switches that allow selected regions of the adjustable polarizer to be configured to serve as vertical-pass linear polarizers, horizontal-pass linear polarizers, or non-polarizing regions), tunable lenses (e.g., liquid crystal tunable lenses, tunable lenses based on electrooptic materials, tunable liquid lenses, microelectromechanical systems tunable lenses, or other tunable lenses), an adjustable color filter (e.g., an adjustable-color-cast light filter that can be adjusted to exhibit different color casts and/or a monochromatic adjustable-intensity light filter that has a single color cast), and/or an adjustable opacity system (e.g., a layer with an adjustable opacity for providing a darkened background if the display is transparent). There may be any suitable number of display pixels in display 26 (e.g., 0-1000, 10-10,000, 1000-1,000,000, 1,000,000 to 10,000,000, more than 1,000,000, fewer than 1,000,000, fewer than 10,000, fewer than 100, etc.).

Apple's patent application 20190331919 that was published today by the U.S. Patent Office was filed back in Q3 2018. Considering that this is a patent application, the timing of such a product to market is unknown at this time.