A Google (Lytro) Patent Application Published by USPTO Last Month Dives into Photo Refocusing Technology
It was reported last year that Google was acquiring Lytro. Then the story shifted to Google acquiring some Lytro patents, more specifically 59 light-field and imaging patents. Lytro had technology that mastered refocusing images after the photos were taken, allowing the bokeh effect to shift to anything in the photo. A simply example of refocusing a photo after it was shot is presented in our cover graphic above.
In the video below about Lytro, going to the 3:50 mark will show you how a user using a PC was able to simply click on any part of a photo using Lytro's software to change the focus of the photo.
This could be a feature that a future Pixel phone camera may be able to deliver.
Last month a Google patent application was published by the US Patent and Trademark Office (USPTO) titled "Capturing light-field Images with Uneven and/or Incomplete Angular Sampling." The patent may be interconnected with what was presented at the top of our report being that in their patent summary the note that the invention will "increase the quality of the depth data, extend the refocusable range of the system."
In the background to the invention they note that light-field cameras may be used to capture a four-dimensional (4D) light-field image, with two spatial dimensions, x and y, and two angular dimensions, u and v. In many light-field cameras, a plenoptic microlens array and a single photosensor, containing a two-dimensional (2D) array of pixels, are used. These plenoptic light-field cameras capture image data that may be much more versatile than traditional two-dimensional image data.
In particular, a light-field image may be processed to create a set of virtual views, in which focus distance, center of perspective, depth-of-field, and/or other virtual-camera parameters are varied within ranges enabled by the data in the four-dimensional light-field image. Further, the light-field data may be analyzed to calculate a depth map and/or analysis information.
One drawback of existing plenoptic cameras is that virtual views have relatively low resolution. In order to capture the four-dimensional light-field, the spatial resolution is decreased significantly in order to increase the resolution of the angular dimensions. In many applications, the output resolution of virtual views may be too low for widespread adoption.
However, depending on the use case, a complete and even sampling in the two angular dimensions may not be required. In many cases, the desired output from the light-field camera may be a combination of high-resolution image data and depth data, often in the form of a depth map or a set of three-dimensional (3D) points (a point cloud). In some use cases, the desired output may be high-resolution depth data.
Google's Patent Summary
Various embodiments of the described system capture light-field images with uneven and/or incomplete angular sampling. Such embodiments may increase spatial and/or output resolution, increase the quality of the depth data, extend the refocusable range of the system, and/or increase the optical baseline. Some of the embodiments utilize novel exit pupil shapes and/or configurations of the microlens array.
Google's patent FIG. 1 below is a diagram of a plenoptic light-field camera; FIGS. 2A to 2E are diagrams of various aspects of the plenoptic light-field camera of FIG. 1.
All three engineers listed on this US patent were previous Lytro engineers/inventors as noted below:
1: Kurt Akeley "Distinguished Engineer" former Lytro CTO
2: Chia-Kai Liang, Senior Staff Software Engineer; former Architect, Computational Photography at Lytro
3: Colvin Pitts Senior Staff Software Engineer at Google; former 10 year Senior Architect as Lytro
Google acquired this 2018 granted patent from Lytro as noted in screenshot below from the USPTO.
For more on this, check out our Patently Mobile report here.