The U.S. Patent and Trademark Office published a patent application from Apple today that reveals a unique lens system design well-suited to augmented reality head-mounted devices that includes a new lens system with 'Lenslets.' This could possibly revolutionize lens systems for AR eyewear.

Apple notes in their patent background that Augmented Reality (AR) or Mixed Reality (MR) systems merge virtual imagery onto a view of the real world. In AR head-mounted devices (HMDs), virtual content is displayed with an image projection system located near the eye. Real-world content can either be viewed directly using an "optical see-through" lens design, or it can be rendered digitally using scene cameras for "video see-through."

Current video see-through HMDs suffer from a shifted perspective of the real-world because the scene cameras are mounted anterior (in front of), superior (above), and/or lateral (to the side) of the user's eyes.

Scene cameras placed lateral to the eyes create a mismatch in Inter-pupillary distance (IPD) compared to the user's eyes. As a result, the user can suffer from double vision, blurred vision, dizziness, headache, nausea, and fatigue. This mismatch can also result in the incorrect perceived distance, and scale, of objects. Scene cameras placed superior or anterior to the eyes can create a perspective that translates with exaggeration to head movement, or incorrect motion parallax. These effects can also cause the user discomfort and this is what Apple's granted patent is out to solve.

Apple's patent filing describes a novel lens system having an entrance pupil at a location matching, or nearly matching, the eye's optical system entrance pupil.

Apple's lens system is significantly thinner than a conventional lens system making it well-suited to head-mounted device (HMD) applications. The system's thinness is achieved by using an array of camera elements.

Each camera element containing camera optics called "lenslets" and a sensor (e.g., the sensor could be an image sensor or a depth sensor), where a lenslet can include one or more lenses (e.g., a compound lens).

Individual camera elements can be arranged such that their optical axes intersect at, or near, the eye's entrance pupil location. In addition, each camera element's field-of-view (FOV) corresponds to a small sector of a large FOV. The FOVs of adjacent camera elements can be non- or slightly-overlapping so that a wide-angle image can be formed by concatenation of the individual camera element images.

Apple's patent FIG. 2 below illustrates a camera system having its individual elements arranged along a spherical surface.

Apple's patent FIG. 6A above illustrates an augmented reality (AR) HMD system #600 in accordance with one or more embodiments includes multiple camera elements (represented by shaped elements) for each eye (e.g., 605L and 605R) arranged along a spherical surface, one or more display elements (#610L and #610R), one or more modules (#615L and #615R) and structure #620 to which the other components may be affixed and which can be mounted onto a user's head. While only five camera elements are illustrated for each eye, more or fewer may be used in any given implementation depending upon the device's intended use and targeted FOV.

Apple's patent FIG. 6B above is another embodiment AR-HMD system #625 includes optic and electronic systems (6051, 6101, and modules 615) affixed to structure (#630) for only a single eye. In the illustrated system, electronic modules #6151 and #615R are shown although both may not be necessary.

Although not mentioned in the body of the patent, Apple's patent claims interestingly state that "the computer readable code to identify one or more objects in the scene comprises computer readable code to perform facial recognition to identify a person in the scene." This could be a major feature of Apple's future eyewear and it's unfortunate that Apple's patent doesn't elaborate on it. If we're lucky, Apple will include more details of this feature in a secondary patent on lenslets in the future.

For more details, review Apple's patent application number 20220146820. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Apple Inventors

Noah Bedard: Prototyping Engineer. He previously worked at Ricoh prototyping novel imaging systems.

Branko Petljanski: Senior Engineer – Camera Electronics (Apple, Munich Germany). He previously worked at Panavision.

Kathrin Berkner: Senior Engineering Manager - Camera Incubation. Incubates user experience, imaging architectures, and technologies to pave the future of camera applications.

Ricardo Motta: Distinguished Engineer for 6 years: Created and led Camera Incubation, a team focused on development of new camera technology. Motta is now working at Google.