On Thursday the US Patent & Trademark Office published a patent application from Apple that relates to AR displays with optical systems that accurately merge display and real-world content with tunable lenses. While the main thrust of the invention relates to a mixed reality headset, Apple states that such a system could have far reaching applications from a Heads-Up display for vehicle windshields through to next-generation contact lenses.

In first describing the issue to solve, Apple notes that wearable electronic devices such as head-mounted devices may include displays for displaying computer-generated content that is overlaid on real-world content. An optical system is used to merge real-world content and display content.

Challenges can arise in providing satisfactory optical systems for merging real-world and display content. If care is not taken, issues may arise with optical quality and other performance characteristics.

Apple's invention relates to a head-mounted device with a display having an optical system through which a user with eyes in eye boxes may view real-world objects. During operation, the optical system may be used to merge real-world images from real-world objects with display images.

A display may produce images in frames. Different objects may be displayed in alternating image frames. The optical system may be adjusted in synchronization with the alternating image frames to display the different objects at different focal planes.

Real-world content may be merged with display content using time-division multiplexing, polarization multiplexing, and/or other arrangements for combining light from real-world objects with light from displays.

Apple's patent FIG. 1 below is a schematic diagram of a head-mounted display device; FIG. 2 is a top view of an illustrative head-mounted device; FIGS. 3A and 3B are cross-sectional views of illustrative optical systems with time interleaving and tunable lenses.

Sensors Galore

The Input-output circuitry #22 of FIG. 1 above may include sensors that could include, for example, three-dimensional sensors (e.g., three-dimensional image sensors such as structured light sensors that emit beams of light and that use two-dimensional digital image sensors to gather image data for three-dimensional images from light spots that are produced when a target is illuminated by the beams of light, binocular three-dimensional image sensors that gather three-dimensional images using two or more cameras in a binocular imaging arrangement, three-dimensional lidar (light detection and ranging) sensors, three-dimensional radio-frequency sensors, or other sensors that gather three-dimensional image data), cameras (e.g., infrared and/or visible digital image sensors), gaze tracking sensors (e.g., a gaze tracking system based on an image sensor and, if desired, a light source that emits one or more beams of light that are tracked using the image sensor after reflecting from a user's eyes), touch sensors, buttons, capacitive proximity sensors, light-based (optical) proximity sensors, other proximity sensors, force sensors, sensors such as contact sensors based on switches, gas sensors, pressure sensors, moisture sensors, magnetic sensors, audio sensors (microphones), ambient light sensors, microphones for gathering voice commands and other audio input, sensors that are configured to gather information on motion, position, and/or orientation (e.g., accelerometers, gyroscopes, compasses, and/or inertial measurement units that include all of these sensors or a subset of one or two of these sensors), radio-frequency sensors that determine the location of other devices (and therefore the relative position of such devices relative to device 10), and/or other sensors.

Apple clarifies that while an HMD is the focus device of this invention, the invention could apply to others in the future such as projection-based systems, heads-up displays (HUDs), vehicle windshields having integrated display capability, windows having integrated display capability, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones/earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop/laptop computers. A head mounted system may have one or more speaker(s) and an integrated opaque display.

Although the patent in general notes smartglasses as another application, it would be way down the road when the lens system could be miniaturized. It is not the main focus of the invention.

To review the patent filing for finer technical details that are many, read the full patent application 20210048674.

Considering that this is a patent application, the timing of such a product to market is unknown at this time.