Today the US Patent & Trademark Office published a patent application from Apple that relates to a next-gen light field panoramic camera system that will eventually be coming to the iPhone, iPad and future HMD. This is important because Google will be delivering such a system to market at some point in the future because of their acquisition of Lytro in 2018. Lytro was one of the leaders in light field imagery relating to photography and VR.  

In the big picture, Apple's invention will deliver a light field panorama system in which a user holding a mobile device performs a gesture to capture images of a scene from different positions. Additional information, such as position and orientation information, may also be captured. The images and information may be processed to determine metadata including the relative positions of the images and depth information for the images. The images and metadata may be stored as a light field panorama.

The light field panorama may be processed by a rendering engine to render different 3D views of the scene to allow a viewer to explore the scene from different positions and angles with six degrees of freedom. Using a rendering and viewing system such as a mobile device or head-mounted display, the viewer may see behind or over objects in the scene, zoom in or out on the scene, or view different parts of the scene.

If you've ever seen such imagery and capability, then you know this will be a killer app for iPhone cameras and future HMD.

Technically, Apple states that in light field photography, a light field camera captures color intensities of light in a scene, and also the direction that the light rays are traveling in space. This contrasts with a conventional camera, which records only light color intensities. One type of light field camera uses an array of micro-lenses placed in front of an image sensor. Multi-camera arrays are another type of light field camera.

Virtual reality (VR) allows users to experience and /or interact with an immersive artificial environment, such that the user feels as if they were physically in that environment. For example, virtual reality systems such as head-mounted displays (HMDs) may display stereoscopic scenes to users in order to create an illusion of depth, and a computer may adjust the scene content in real-time to provide the illusion of the user moving within the scene.

When the user views images through a virtual reality system, the user may thus feel as if they are moving within a scene from a first-person point of view. Virtual reality systems may be utilized to provide an interactive user experience for multiple applications.

Apple's invention provides various embodiments of methods and apparatus for capturing, processing, and rendering light field panoramas.

In embodiments of a light field panorama system, a user holding a mobile device that includes a camera, such as a smartphone, tablet, or pad device, performs a gesture to move the camera in front of a scene of interest to capture a set of digital images of the scene from different positions. 

Apple's patent FIG. 2 below graphically illustrates components of a light field panorama system.

Apple's patent FIGS. 4A through 4F above illustrate non-limiting, example gestures on an iPhone that may be used to capture frames for generating a light field panorama. Apple's patent FIGS. 5A and 5B below graphically illustrate viewing a light field panorama using a hand-held mobile device such as a smartphone or pad device.

Apple's patent filing further notes that additional information, for example, position and orientation information from motion and position sensing technology of the device, may also be captured with the images.

The captured images and information may be processed to determine metadata including the relative camera positions of the images with respect to the scene and depth and geometry information for content of the scene captured in the images. The images and metadata may be collectively referred to as a light field panorama.

The captured scene represented by the light field panorama may be explored by a viewer using a rendering and viewing system on an HMD, a mobile device such as a smartphone, tablet, or pad device, or on a computer system. The light field panorama data (images and metadata) for the scene may be processed by a rendering engine to render different 3D views of the scene to allow the viewer to explore the scene from different positions and angles with six degrees of freedom.

Using the rendering and viewing system, the viewer may change their viewing position and angle to see behind or over objects in the scene, zoom in or out on the scene, or view different parts of the scene.

Thus, the light field panorama allows a viewer to explore a scene with six degrees of freedom (6DOF), meaning the viewer can rotate with the content as well as translate in different directions. By contrast, a typical 360 panorama (or photo sphere) only allows three degrees of freedom in the rendering, meaning that the viewer can only rotate their head but cannot translate through the content as they can when exploring the light field panorama.

Embodiments may, for example, allow the viewer to experience the captured wide-angle content of a scene in immersive virtual reality, for example via an HMD.

The image that is captured is 'parallax' aware in that when the image is rendered in virtual reality, objects in the scene will move properly according to their position in the world and the viewer's relative position to them. In addition, the image content appears photographically realistic compared to renderings of computer-generated content that are typically viewed in virtual reality systems.

Lastly, Apple's patent FIG. 7 below illustrates a real-time and post-processing architecture for a light field panorama system.  

Apple's patent application that was published today by the U.S. Patent Office was filed back in Q3 2019. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

For the record, Patently Apple covered an IP report back in June 2017 titled "Apple Wins Key Patent for Light Field Cameras that could deliver an Immersive AR Experience for Macs, iDevices and Headset." The patent was originally filed in 2016. So Apple has been working on this project for at least 3.5 years. 

Some of Apple's Inventors

Dan Lelescu, Computational vision; Tobias Rick, Video Engineering; Brett Miller, Engineering Manage, Camera Incubation; Gary Vondran, Imaging Scientist, Camera Architecture