In late February Patently Apple posted a report titled "A Euro Patent Reveals Apple is working on an 8K Foveated Micro-Display for a Head-up Display, iDevice and beyond." Apple is in a race of sorts with Qualcomm their mortal enemy regarding the creation of a platform supporting AR Glasses and Heads-up Display systems. Apple's first patent filing for a Foveated Display described using "micro-displays" and a gaze control system. This week a second Foveated Display patent application was published in Europe with greater detail surrounding the system and a tiny admission that the high-end display will be used for gaming applications.

Apple's invention relates generally to devices with displays, and, more particularly, to devices with foveated displays.

Displays may be incorporated into devices such as head-mounted devices and/or AR glasses and the like. It may be desirable to provide users with immersive content using the display. For example, it may be desirable for a user's entire field-of-view to be filled with content on a display.

Apple notes in their patent background that "If care is not taken, however, images viewed by a user will not be satisfactory. High-resolution images may require excessive bandwidth and may therefore be difficult or impossible to display effectively at satisfactory frame rates. The bandwidth used in conveying image data to a display may be reduced by lowering image resolution, but excessively reduced image resolution may degrade image quality more than desired." This is where Apple's invention steps into the picture.

The focus of the invention is delivering a foveated display for AR Glasses and/or a head-mounted display so that a user's entire field-of-view could be filled with content on the display at the highest quality.

The second focus of the invention revolves around a gaze detection system that gathers information on a user's point of gaze on the display. Based on the point-of-gaze information, control circuitry in the electronic device may produce image data for an image with multiple resolutions. A full-resolution area of the image overlaps the point of gaze. Lower resolution image areas are located in peripheral regions of the image.

The display has a pixel array. Display driver circuitry may be used to display the image using the pixel array. The pixel array may include rows and columns of pixels. Data lines may be used to supply data to columns of pixels in accordance with row selection signals supplied to rows of pixels. Apple's patent FIG. 6 below is a diagram of a pixel array and associated display driver circuitry.

The display driver circuitry may have row selection circuitry for supplying the row selection signals to the pixel array and may have column expander circuitry for routing data to the data lines of the pixel array. The row selection circuitry and column expander circuitry may be responsive to a resolution mode selection signal.

The column expander circuitry may have a bus-line buffer that receives the image data from the control circuitry on a first number of data input lines. The bus-line buffer may provide the received image data to an expanded number of bus-line buffer output lines. The column expander circuitry may also have a line buffer that receives the image data from the bus-line buffer output lines. The line buffer may supply image data to a third number of data lines in the pixel array.

A schematic diagram of an illustrative head-mounted display or AR Glasses may be provided with a foveated display arrangement as shown in Apple's patent FIG. 1 below.

Apple later clarifies and describes that a head-mounted display is "a pair of augmented reality glasses that is worn on the head of a user may be used to provide a user with computer-generated content that is overlaid on top of real-world content."

High-End AR Glasses Face Camera

According to Apple, the real-world content may be viewed directly by a user (e.g., by observing real-world objects through an optical coupler in a display system that merges light from real-world objects with light from a display). Configurations in which images of real-world objects are captured by a forward-facing camera and displayed for a user on a display may also be used.

More Details on the Point of Gaze System

In electronic devices such as head-mounted display devices, it may be desirable to display images for users over a wide angle of view. Displays that cover wide angles of view at high resolutions may consume relatively large amounts of image data and may therefore impose bandwidth burdens on electronic devices such as head-mounted displays. These bandwidth burdens may be reduced by using a display scheme in which high resolution images are displayed in alignment with the user's current point of gaze and in which low resolution images are displayed in the user's peripheral vision. Display schemes such as these may sometimes be referred to as foveated display schemes.

An illustrative example of a foveated image being displayed on pixel array #30 (FIG. 1) of display #26 is shown in FIG. 2. In the example of FIG. 2, the display has a rectangular outline. The size of the display may be selected to be sufficiently large to cover most or all of a user's field of view.

Based on gaze tracking information from gaze tracking system #62, graphics processing unit #22 may determine that a user's current point of gaze is located at the point of gaze PG (e.g., in the upper right corner of the display shown in the example of FIG. 2).

Based on this location, the graphics processing unit may render image data in full resolution for full­ resolution area xl of FIG. 2 (an area that overlaps PG). Peripheral image data (e.g., image data for region x8 of FIG. 2) may be rendered at a reduced resolution (e.g., 1/8 of the full resolution). Intermediate areas that lie between full resolution area xl and reduced resolution area x8 may be rendered at 1,4 resolution (see, e.g., the x4 area of the display) and at 1h resolution (see, e.g., the x2 area of the display). In general, any suitable number of different resolutions may be used in rendering image data for the display in the device / AR glasses. The use of four different areas with four respective different resolutions in the example of FIG. 2 is illustrative.

As the user follows visible content on the display, the point-of-gaze location PG will shift to different regions on the display. The graphics processing unit may use this information to adjust the locations of the high resolution and lower resolution areas for which image data is being rendered with different resolutions.

To ensure that display driver circuitry #28 is informed of which resolution applies in each portion of the display for a given image, the graphics processing unit and/or control circuitry #50 may supply the display driver circuitry with information on the boundaries of regions x8, x4, x2, and xl (e.g., gaze tracking system information such as point of gaze PG or more processed information such as information on the boundary locations for regions x8, x4, x2, and xl that is derived from point of gaze PG).

During the operations of block #70 of FIG. 3 below, the control circuitry may use gaze tracking system to gather information on the user's point-of-gaze. The point-of-gaze (see, e.g., point of gaze PG of FIG. 2) may be located at a particular coordinate (X,Y) on the display. As the user's gaze moves (e.g., to track on-screen objects on display 26), the location of point of gaze PG can be updated dynamically.

Gaming Apps on AR Glasses

According to Apple, "A source of content (e.g., a computer program such as a game application or other application running on the control circuitry) may produce content for viewing on the display. To avoid overburdening the circuitry of the AR Glasses, the graphics processing unit may generate foveated display image data during the operations of block 72 of FIG. 3 above.

The foveated data includes high resolution (full resolution) data for a full-resolution region of the display (e.g., the xl region of FIG. 2) and image data of progressively lower resolutions for lower resolution regions (e.g., regions x2, x4, and x8 of FIG. 2).

By rendering images with different resolutions in different areas, the amount of image data that is associated with the content to be displayed on the display can be significantly reduced (e.g., in comparison to rendering the entire image on display 26 at full resolution).

This also lowers the rendering burden on the graphics processing unit and lowers the amount of data bus bandwidth consumed in transferring the rendered image data from the graphics processing unit to the display driver circuitry over one or more data busses between the graphics processing unit and the display driver circuitry.

Apple's patent application was filed in Europe on January 18, 2018 and published on Thursday August 02, 2018. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Patently Apple presents a detailed summary of patent applications and/or granted patents with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.