Earlier this week Patently Apple covered two Apple patent applications relating to their future mixed reality headset. The first invention covered an elaborate adjustable headband system that could automatically adjust on the user's head depending on what content was being viewed. The second invention covered a thermal regulation system so that the user's face inside the headset could be kept cooler.

Today Patently Apple discovered a third headset related patent application filed in Europe this week covering a head mounted display device with an adaptive display. Such a display is specifically designed  to (a) assist a user's eyes adjust from the real world's lighting going into the dark environment of a headset much quicker, and (b) avoid eye dazzle which is experienced when a user exits the dark environment of a headset to the brighter exterior environment.

Once again this latest round of headset patent filings go to show how Apple takes their time when developing a potential new product by taking a holistic view of the product from hardware through to software so as to add value to the end product that will distinguish it from the competitions.

The Problem to Solve: Apple begins by noting that if care is not taken, a head-mounted display device may be cumbersome and tiring to wear.  Firstly, the images on the display may appear too dark and washed out when the user first puts the head­ mounted device on their head.  Secondly, the user may experience dazzle or discomfort when transitioning out of a virtual reality viewing experience. The dynamic range of a head-mounted display may be perceived as insufficient depending on the adaptation state of the user's eyes. This is the problem that Apple's invention is to remedy.

Head Mounted Display Device with an Adaptive Display

Apple's invention generally covers an electronic device configured to be worn by a user that will display all manner of content including virtual reality. More specifically, the invention addresses the problem presented in Apple's introduction.

One of the key aspects in overcoming the problems starts with the headset's control circuitry that estimates a brightness adaptation state of the user that is wearing the electronic device.  The control circuitry may adjust a brightness of the display based on the user's adaptation state. 

The control circuitry may include, for example, optimizing a brightness range of the display for the current adaptation state of the user, adjusting a brightness range of the display to have a desired effect on the user's adaptation state (e.g., to help "guide" the user's current adaptation state to a different adaptation state), adjusting a brightness range at certain periods of time to boost the perceived dynamic range of the display, adjusting brightness in some regions of an image differently than in other regions of an image to account for the user's adaptation state, and/or taking other actions based on the estimated adaptation state of the user.

The control circuitry may gather ambient light information from an ambient light sensor, may gather physiological attributes of a user such as blink rate, pupil size, and eye openness from a camera, and may gather gaze position information from gaze detection circuitry.

Apple further notes that Gaze position may be combined with image data (e.g., frames of display data that are being displayed on the display) to determine the average pixel luminance at a location on the display that aligns with the user's gaze position.  Based on this information, the control circuitry may determine a brightness adaptation state of the user.

When a user first turns on the display of the electronic device, the user may be bright­ adapted (due to the ambient light in the room where the electronic device is located).  The control circuitry may account for this adaptation state by starting at a first brightness level and gradually reducing the brightness of the display to a second brightness level.  By the time the second brightness level is reached, the user may be dark-adapted.

When a video ends or a user exits a video playing mode and enters a home screen on the display, the control circuitry may increase the brightness of the display gradually to increase the user's adaptation level.  This helps avoid dazzle or discomfort when the user takes off the electronic device and is greeted with a brightly lit room.

The control circuitry may also use adaptation state information to increase the perceived dynamic range of the display and to selectively adjust the brightness of portions of an image based on where the user is looking.

Apple's patent FIG. 1 is an overview of the technology behind a HMD Device supporting an adaptive display system providing VR content that may be fed by an iDevice or Mac via wired or wireless paths.

The control circuitry is like the brains of the system and controls everything form storage with an SSD, GPU, baseband processors and so forth. Control circuitry (#42) may use display system (#40) to display visual content such as virtual reality and content that's pre-recorded video for a movie or other media, or other images.

The key parts of the control circuitry include a content generator, tone mapping and adaptation state circuitry.

For instance, a luminance value mapping circuitry such as tone mapping circuitry (#14) may be used to provide content generators with tone mapping parameters indicating how the content generators should map content luminance values to display luminance values

When the human visual system is bright-adapted (i.e., adapted to bright light), the eye is less sensitive.  In contrast, when the human visual system is dark-adapted (i.e., adapted to low light), the eye is more sensitive.

If care is not taken, a user that is adapted to bright ambient light before using a head­ mounted display may initially perceive the images on the display as dim and washed out until the user's eyes adapt to the darkness of the head-mounted display.  Similarly, a user that is adapted to the darkness of a head-mounted display may experience dazzle and discomfort when the user removes the head-mounted display and is faced with bright ambient light.

In patent FIG. 8 above we're able to illustrative image #74 being displayed on the HMD's display (#40).  In the image, region #70 may be mapped using different tone mapping parameters than surrounding region (#72). Region (#70) may be an area of the image that the user's eyes are focused on.  Gaze tracking circuitry (e.g., camera #24) may be used to track a user's gaze, determine a direction of view, and determine which portion of the image is at the center of the user's direction of view.

Apple's patent FIG. 3 below is a diagram showing how a tone mapping circuitry may use information about a user's adaptation state to implement content-luminance-to-display-luminance mappings.

Apple's patent application that was published this week by the European Patent Office was originally filed in Europe back in August 2018. Considering that this is a patent application, the timing of such a product feature to market is unknown at this time.

Apple began their R&D headset work more than a decade ago. In respect to patents, Patently Apple's first report covered a 2008 invention that was granted a patent in September 2009. Apple's work has advanced over the years and you could find many other headset patents in our patent archives titled "Face ID / HMD / Smartglasses here.

Rumor of the Week

It's not uncommon when new Apple headset patents surface in the U.S. or Europe, that rumors surface making predictions.

It was interesting to note this week that the Chinese tech site Economic Daily reported that Guo Minghao, an analyst at Tianfeng International Securities, forecasted that Apple's first-generation head-mounted AR device would be mass-produced in the fourth quarter of this year to the second quarter of next year, which would mark an important milestone in the commercialization of augmented reality.

The report further noted that "Apple is actively deploying augmented reality areas. Foreign technology website Patently Apple previously reported that a European patent shows that Apple is deeply cultivating augmented reality technology applications, which can be applied to the windshield of automobiles in the future, and can also be applied to head-up displays." You could read the full Chinese report here. The link to the rumor was first discovered by MacRumors.

One should be skeptical of rumors trying to predict the timing of Apple's headset to market until more substantial evidence arises.

As Apple's iPhone sales possibly slow further over time, a high-end headset used as an iPhone accessory could positively boost revenue in general and iPhone revenue specifically if Apple introduces a new key technology in a future iPhone that would be required to use the new headset.

About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.