The U.S. Patent and Trademark Office officially published a series of 62 newly granted patents for Apple Inc. today. In this particular report we cover two Apple patents relating to future Mixed Reality Glasses. The first covers coordinating the camera and display system for Augmented Reality overlays and the second covers eye-gaze controls.    

Smartglasses with Coordinated Camera and Display Operation

Apple's granted patent relates to a head-mounted device such as a pair of mixed reality glasses that is worn on the head of a user may have a camera such as an outwardly facing camera. During operation, the camera can capture a moving image of the real-world environment surrounding a user so that control circuitry in the electronic device can display the real-world content for the user. The user may also be provided a user with computer-generated content (sometimes referred to as virtual content).

The head-mounted device may operate in a virtual reality mode in which virtual reality (computer-generated) content is displayed for a user. In this mode of operation, real-world content captured by the camera is not displayed. The head-mounted device may also operate in a mixed reality mode.

In the mixed reality mode, virtual content (sometimes referred to as non-real-world content) is overlaid on the real-world content captured by the camera. The virtual content may, for example, be text, graphics, moving images, and/or other content that is displayed over portions of the real-world content that is displayed.

To ensure that content is displayed satisfactorily, display and camera settings may be adjusted dynamically when transitioning between virtual reality and mixed reality modes.

A camera can capture images at a camera frame rate (sometimes referred to as an image capture frame rate) and a display can display content (e.g., a moving image) at a display frame rate. When transitioning between virtual and mixed reality modes, the camera and display frame rates can be adjusted dynamically and operations such as camera reconfiguration operations may be performed at times that minimize disruptions to system operation.

Apple's patent FIG. 1 below is an overview of a system supporting a pair of glasses; In FIG. 2 we see a support structure #16, which may sometimes be referred to as a housing, may be configured to form a frame of a pair of glasses.

Apple's granted patent 10,748,340 was originally filed in Q2 2018 and published today by the US Patent and Trademark Office.

Glasses: Image Enhancement & Gaze Tracking

Apple's second granted patent 10,747,312 covers mixed realty glasses with gaze tracking. Mixed reality glasses may have cameras and displays. The cameras can capture images of the real world. Content such as text and graphics can then be overlaid on the real-world images to produce mixed-reality images.

Apple notes that it can be challenging to present mixed reality content to a user. The presentation of mixed-reality images to a user may, for example, be disrupted by head and eye movement. Users may have different visual capabilities. If care is not taken, mixed-reality images will be presented that cause disorientation or motion sickness and that make it difficult for a user to identify items of interest.

An electronic device such as glasses may have a display and a camera. The camera may be used to capture a real-world image. The real-world image can be displayed on the display for a user.

Control circuitry in the device can gather information on the user's point of gaze using a gaze tracking system and other sensors. The control circuitry can also gather information on the real-world image such as information on content, motion, and other image attributes by analyzing the real-world image. User vision information such as user visual acuity, contrast sensitivity, field of view, and geometrical distortions can be provided to the control circuitry.

Input-output circuitry can be used by the control circuitry to gather user input such as user preferences and user mode selection commands and can be used to gather location information and other input

Based on the point-of-gaze information and/or other gathered information, the control circuitry can display supplemental information on the real-world image on the display. The supplemental information can include augmentations such as icons, text labels, and other computer-generated text and graphics overlaid on the real-world image and can include enhanced image content such as magnified portions of the real-world image. If desired, the real-world image may be globally or locally enhanced by adjusting contrast, altering image color, enhancing edges, adjusting for geometrical vision distortion, and performing other image enhancements on the real-world image.

Apple's patent FIG. 2 below is a diagram showing how gaze tracking system #16 which gathers eye position information on a user's eye. In a typical scenario, gaze tracking system may include components for simultaneously monitoring two eyes; FIG. 7 is a flow chart of illustrative operations involved in providing a user with mixed reality content

In block #84 we see that it relates to those with prescription glasses. Apple notes that "User data gathered during the operations of block #84 (e.g., a user's visual impairment details) may be taken into account during the operations of block #90. For example, if the user's vision is characterized by a geometric distortion, a compensating geometric distortion can be applied in displaying the image on display  #14. If the user has low visual spatial acuity, content can be magnified appropriately.

Apple's granted patent 10,747,312 was originally filed in Q1 2019 and published today by the US Patent and Trademark Office.