On Thursday the US Patent & Trademark Office published a patent application from Apple that relates to a new peripheral device designed to work with their future Mixed Reality Head Mounted Display (HMD) device. More specifically, the peripheral device is a lighting control device. The controller is configured to determine a target lighting condition in a room relative to a current lighting condition in the room. The controller is also configured to generate a control signal with instructions to adjust an ambient lighting peripheral in the room based on the determined target lighting condition.

In Apple's patent background they note that in a computer-generated reality (CGR) scenario, a head-mounted device is used to cover the eyes of its user, and CGR content is displayed by the head-mounted device. One of the challenges for successfully commercializing CGR devices relates to providing a quality experience to the user. Merely providing a head-mounted device and CGR content is not effective if the quality of the CGR experience is deficient.

One option to provide CGR content involves the use of one or more cameras on the head-mounted device (e.g., a camera captures images of what the user would see without the head-mounted device).

In this scenario, captured camera images can be combined with CGR content to display what has been referred to as "mixed reality" content on the head-mounted device.

Improving the quality of CGR content, including mixed reality content, is not a trivial task. There are conflicting limitations related to cameras, content generation, display quality, processing speed, and system cost that have not been adequately resolved.

Apple's invention covers an apparatus that comprises a transceiver and a controller in communication with the transceiver.

The controller is configured to determine a target lighting condition in a room relative to a current lighting condition in the room. The controller is also configured to generate a control signal with instructions to adjust an ambient lighting peripheral in the room based on the determined target lighting condition. The control signal is provided to the transceiver for transmission.

In accordance with at least one embodiment, a system comprises a head-mounted device configured to capture camera image data and to display computer-generated reality content.

The system also comprises at least one ambient lighting peripheral.

The system also comprises a controller configured to determine a target ambient lighting condition relative to a current ambient lighting condition based on camera image data captured by the HMD. The controller is configured to provide control signals to adjust the at least one ambient lighting peripheral based on the target ambient lighting condition.

A person, using the system, indirectly views the physical environment by way of the images or video of the physical environment, and perceives the virtual objects superimposed over the physical environment. In Apple's patent, this process is called a "pass-through video," meaning a system uses one or more image sensor(s) to capture images of the physical environment, and uses those images in presenting the AR environment on the opaque display.

Further alternatively, the HMD system may have a projection system that projects virtual objects into the physical environment, for example, as a hologram or on a physical surface, so that a person, using the system, perceives the virtual objects superimposed over the physical environment.

Apple's ambient light control technique enables adjusting the ambient lighting conditions to reflect the impact that a virtual object should have on the real environment.

For example, if CGR content includes a virtual glowing blue orb, a user would expect that the real environment near that orb would look blue. As the orb moves, the blue glow in the room should move with it.

Rather than performing computationally-expensive image processing to provide CGR content involving the blue orb, the disclosed ambient light control technique enables changes to the color of a light bulb in the room near the orb to be blue.

Apple's patent FIG. 1 below is a block diagram showing a system; FIG. 2 is a block diagram showing an ambient lighting control scenario.

In patent FIG. 1 above, the 14 ambient lighting peripherals 120A-120N are in the same room or space as the HMD such that ambient lighting controlled by the ambient lighting peripherals 120A-120N affect the appearance of CGR content displayed by the HMD.

In other examples, less than all of the ambient lighting peripherals 120A-120N are in the same room or space as the HMD (e.g., the ambient lighting peripherals 120A-120N are distributed throughout different rooms of a house or office space).

In addition, patent FIG. 1 above illustrates that the system (#100) optionally includes a base station (#112) with a controller (#114) and a wireless transceiver (#116). The base station and the HMD exchange information (e.g., camera images, CGR content, current ambient lighting condition information, target criteria, target ambient lighting condition information, control signals) via a wireless communication channel (#118A) while performing the ambient lighting control process.

Apple's patent application 20200103656 that was published Thursday by the U.S. Patent Office was filed back in Q3 2019. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Apple Inventors

James McIlree: Kernel Engineer

Sam Iglesias: Senior Software Engineer, AR/VR

Tim Oriol: Senior Software Engineering Manager, Technology Development Group at Apple; Specialties: API Design, System Architecture, AR/VR, UI, UX, Graphics, Metal & iOS.

In Young Yang: Content Director