Last Tuesday Patently Apple posted a granted patent report titled "Apple won a Wild patent for an Augmented Reality Light System that projects AR content onto tables, walls+ without a Headset." One of the images from that patent is presented below.

Today the U.S. Patent and Trademark Office officially granted Apple a major patent that relates to this project described last week with far more detail. Apple notes that a spatial light system may be used to project image content onto surfaces of a room. The system may include two or more projection units for emitting light onto surfaces within a room. The system may include a controller and two or more projection units, each module including an LED array, an array of light pipes, and a condenser lens.

Two or more projection units may be connected to a flexible strip that provides power and data (e.g., serial data) to the projection units. Two or more flex strips may be connected together. The flex strips provide a serially-connected, flexible modular architecture for spatial light systems that allow the projection units to be conformed to a variety of configurations and shapes. 

Apple's patent FIG. 1G Below illustrates a projection unit mounted on a flexible strip.

Various embodiments of methods and apparatus for emitting light including light representing augmented reality (AR) content into environments such as rooms are described. Embodiments of a spatial light system are described that may, for example, be used to project AR content onto one or more surfaces of a room. In some embodiments, a spatial light system may include two or more light-emitting diode (LED) projection units for emitting light representing AR content onto surfaces within a room, onto object(s) within a room, or onto surfaces in a portion of a room. In some embodiments, a spatial light system may also be configured to emit diffuse light to illuminate a room, object(s) within a room, or a portion of a room when not emitting light representing AR content.

In some embodiments, a spatial light system may include a controller comprising one or more processors and one or more projection units. Embodiments of projection units are described that each include an active matrix LED array, an array of light pipes, and condensing optics (referred to as a condenser lens). The LED array may, for example, be an array of individually addressable LEDs. The LEDs may include white light-emitting LEDs and color-emitting LEDs. While embodiments are generally described in which projection units include light pipe arrays, other combination of light collection or light shaping optical elements may be used.

In some embodiments, two or more projection units may be connected to a flexible strip (referred to as a flex strip) that provides power and data (e.g., serial data) to the projection units. Two or more flex strips may be connected together. The flex strips of projection units provide a serially-connected, flexible modular architecture for spatial light systems that allow the projection units to be conformed to a variety of configurations and shapes. However, in some embodiments, the projection units may be mounted to a fixed surface that includes connections for power and/or data for the modules. In some embodiments, one or more other modules such as sensor modules (depth sensors, light sensors, motion sensors, cameras, etc.) may also be connected to the flex strips in addition to the projection units, for example to provide depth information for surfaces and object in the room, or to detect presence and motion of persons in the room.

In some embodiments, components of a spatial light system including but not limited to the projection units may be packaged or contained within an enclosure or “bulb-like” structure of a transparent or semi-transparent material (e.g., glass or plastic), as shown below.

Apple's patent FIG. 6 below illustrates a spatial light system that is "bulb-like;"  FIG. 8 illustrates components and operation of an example spatial light system that receives room information from an external device such as an iPhone, iPad, Mac, Television, HMDs and more.

Apple's patent FIGS. 18, 19 and 20 above illustrate a use case where a spatial light system is used to selectively illuminate surfaces in a room. In FIG. 18, the ceiling is illuminated. In FIG. 19, the floor is illuminated. In FIG. 20, the walls are illuminated; FIG. 22 shows a use case where a spatial light system is used to provide accent lighting within a room.

Apple further notes that in some embodiments, a spatial light system may be connected to electrical power, for example via a conventional light socket or via a conventional power plug. Embodiments of a spatial light systems as described herein, may, for example, be hung from a ceiling, or mounted on floor or table stands as presented in patent FIGS. 2A-C below.

Further, in some embodiments, a spatial light system may include a controller or control system that may, for example, provide power and video signals to the LED projection units. In some embodiments, a spatial light system may also include wireless technology that enables wireless communications between the spatial light system and other devices, such as an AR/VR system (e.g., an AR/VR headset or glasses), a mobile multipurpose device such as a smartphone, pad or tablet device, a smart TV, or a computer system. In some embodiments, a spatial light system may instead or also include technology that enables wired connection between the spatial light system and other devices.

In some embodiments, a spatial light system may include integrated sensors to detect objects and surfaces (e.g., people and specular surfaces such as mirrors or windows) within a room, and/or to detect light levels within a room.

The sensors may also determine depth of objects or surfaces, and types of objects or surfaces within a room. In some embodiments, at least one of the sensors may use infrared (IR) light to detect objects or surfaces. Information about objects and surfaces within a room collected by the sensors may be fed back to a controller of the spatial light system, which may use the information in controlling operation of the projection units of the spatial light system.

A spatial light system may include a controller comprising one or more processors to which projection units are connected, for example via flex strips. The controller may drive or control the projection units on the flex strips to emit light into a room according to image content and room information. The room information may include, but is not limited to, depth information for surfaces in the room, lighting information for the room, specular information for surfaces or objects in the room, location and identification information for objects in the room, location, motion, and orientation information for persons in the room, or in general any mapping information that can be obtained for a room and its current contents.

For example, one or more sensors may collect lighting information for existing artificial and natural light sources in a room, such as lamps, televisions, and windows. The room information may, for example, be used to determine depths at which “pixels” projected by the projection units are to be focused, objects or surfaces to be illuminated or not illuminated, and so on.

In some embodiments, the room information may be obtained from an external device, for example via a wireless connection to the system. In these embodiments, the system controller may be relatively simple, projecting images onto surfaces based on the room information obtained from an external device.

In some embodiments, the room information may be generated by the system controller from data obtained from one or more sensor modules of the system, for example sensor modules coupled to the flex strips on which the projection units are coupled. In these embodiments, the system controller may be relatively more complex to provide the needed processing power.

Image content to be projected by the spatial light system may be obtained from an external device, for example via a wireless connection, or alternatively may be generated based on image information stored locally in memory of the system. The spatial light system may, for example, project illuminating light, AR imagery, patterns, or other image content onto the floor, walls, and/or ceiling of a room based on the image content and room information. However, the system may also intelligently light or augment particular objects found within a room, and/or intelligently not illuminate certain surfaces or objects such as windows or television screens, or persons, within a room, based on the room information obtained from an external device or generated by the system controller from data collected by sensors.

In some embodiments, the room information may be used by the spatial light system to detect surfaces, objects, and motion within a room, and may also be used to identify particular objects including but not limited to specular surfaces such as mirrors, windows, and television screens, and persons within the room. This information may be used to adjust projection of light from the spatial light system.

For example, upon detecting a person within the room, the spatial light system may lower the intensity of the light projected towards that person by one or more of the projection units to avoid “dazzling” the person with intense light.

As another example, the spatial light system may adjust one or more of the projection units to focus the projected imagery at different depths as detected by the sensors.

As yet another example, the spatial light system may lower the intensity of projected light, or not project light, towards a reflective surface to avoid unwanted reflective light from the reflective surface. As still yet another example, the spatial light system may project light towards an identified object or objects within the room to intentionally highlight or “spotlight” those objects.

As still yet another example, a spatial light system may include one or more sensors that allow the system to track moving objects or persons in a room; the spatial light system may then project light with respect to the tracked object or person to augment that object or person, for example to project light in front of a moving toy car to provide the appearance of headlights of the car, or to “spotlight” a moving person in a room.

In some embodiments, the system may project 2D images of AR or other image content onto surfaces in the room, for example at least portions of the walls, ceiling, and floors, or onto objects within the room.

In some embodiments, the image content projected by the system may be grayscale or color image content. Projected AR content may, for example, display swaying shadows of trees on the walls and ceiling, images of animals, birds flying, clouds, or of rain falling. Instead of or in addition to projecting image content in motion, still image content may be projected.

In some embodiments, the system may also be configured to emit diffuse light to illuminate a room when not emitting light representing AR or other image content.

In some embodiments, the system may emit spatial light in combination with other devices, for example to provide the expansion of television content. In some embodiments, the system may emit light to provide a low resolution version of a virtual environment that a person in the room is experiencing in virtual reality (e.g., using a headset or head-mounted device (HMD)) to give context to other people in the room without virtual reality headsets.

In some embodiments, the system may emit light to complement a mixed reality environment being experienced by a person using an augmented reality headset by using lighting to improve contrast and colors viewed in the headset.

Apple's Inventors

• Rongrong Liu: Imaging Scientist | PhD in Optical Imaging
• Blake  Coughenour: Optical Tech Lead at Apple
• Bacim de Araujo e Silva; Felipe: UX Prototyping Engineer
• Noah Bedard: Prototyping Engineer
• Bosheng Zhang: Prototype Engineer, Camera Incubation