Color Balancing and Correction

As an overview principle, Apple states that color balancing may be thought of as the global adjustment of the colors in an image. One goal of color balancing is to render specific colors, e.g., neutral white, as accurately as possible to the way the color appeared in the actual physical scene from which the image was captured. In the case of rendering neutral white colors correctly, the process is often referred to as "white balancing." Most digital cameras base their color balancing and color correction decisions at least in part on the type of scene illuminant.

For example, the color of a white sheet of paper will appear differently under fluorescent lighting than it will in direct sunlight. The type of color correction to be performed may be specified manually by a user of the digital camera who knows the scene illuminant for the captured image, or may be set programmatically using one or more of a variety of automatic white balance (AWB) algorithms.

Beyond the overview, Apple then defines the specific techniques they employ to advance color balancing and color correction in their iDevice cameras beyond prior art. Patently Apple has painstakingly crafted the patent graphics below so as to clearly capture and show you where Apple's advancements occur in the process without having to get overly technical.

Patent One: Transformations and White Point Constraint Solutions for a Novel Chromaticity Space

Apple's first patent application generally relates to the field of image processing. More particularly, but not by way of limitation, it relates to techniques for creating a novel chromaticity space that may be used as a framework to perform color balancing on images captured by a variety of different image sensors.

More specifically, Apple's invention is for a novel chromaticity space that may be used as a framework to implement an auto-white balance solution or other color image processing solutions that take advantage of the particular properties of the novel chromaticity space. The chromaticity space may be defined by using a series of mathematical transformations having parameters that are optimized to adapt to specific sensors' spectral sensitivities. The unique properties of the novel chromaticity space provide a conscious white point constraining strategy with clear physical meaning. In this chromaticity space, the ranges of possible white points under different kinds of lighting conditions can be defined by polygons. Because of the physical meaning the chromaticity space, the projection that is needed to bring an initially "out-of-bounds" white point back into the polygon also carries physical meaning, making the definition of projection behavior and its consequences conceptually clean and predictable.

In the first graphic presented below, you'll note that we present you with the prior art of how the color balancing process used to function. This applies to all four patents of this report. Then notice that there is a new process illustrated below the prior art in the first three patents positioned between the steps relating to "RBG Raw Data" and "White Point Calculation and Balancing." In each of the first three patents, Apple illustrates an extra step that takes place in the process that uniquely advances color balancing.

In each of the four patents below, the FIG. 2 represents a unique added step in the imaging process that we highlight in yellow. In each of the patents below, the FIG. 3 represents an exploded view of what is actually occuring in that new step that Apple has added to advance color balancing and/or correcting.  

Patent Two: Alleviating Dominant Color Failure in Automatic White Balance Using Histogram Trimming

Apple's second patent application generally relates to the field of color balancing. More particularly, but not by way of limitation, it relates to techniques for alleviating problems associated with dominant color failures in auto white balance (AWB) algorithms.

More specifically, Apple's invention relates to methods, devices and computer readable media for implementing novel dominant color alleviation techniques for color balancing are described. The techniques take advantage of unique properties of 2D image data histograms accumulated in a chromaticity space, along with other factors such as estimated scene lux and knowledge of plausible scene illuminant white point values within the chromaticity space. The accumulated 2D image data histograms may be refined and "trimmed," such that the resultant image data passed to an auto white balance solution has much less influence from the dominant colors in the image, even those that overlap the plausible scene illuminant color region. The described techniques provide for white point estimates that are much less prone to dominant color failures.  

Patent Three: Use of Noise-Optimized Selection Criteria to Calculate Scene White Points

Apple's third patent application generally relates to the field of color balancing. More particularly, but not by way of limitation, it relates to techniques for improving the performance of auto white balance (AWB) algorithms by using noise-optimized selection criteria.

More specifically, Apple's invention relates to methods, devices and computer readable media for implementing a "selective gray world" approach for color balancing are described. The disclosed techniques involve the use of noise-optimized selection criteria and, more specifically, in some embodiments, the interpolation between corresponding values in noise-optimized weighting tables when calculating white balance gains. Estimated scene lux levels may provide a valuable indicator of expected scene noise levels. The image processing techniques described herein may be executed by an image capture device or a general purpose processor (e.g., personal computer) executing a user-level software application. The described color balancing techniques may be implemented by dedicated or general purpose hardware, general application software, or a combination of software and hardware in a computer system.

Patent Four: Multi-Illuminant Color Matrix Representation and Interpolation Based on Estimated White Points

Apple's fourth and final patent camera related patent application generally relates to the field of color correction. More particularly, but not by way of limitation, it relates to techniques for deriving improved approximation and interpolation of color correction matrices based on using a multi-illuminant color matrix representation and interpolation based on estimated white points.

More specifically, Apple's invention relates to devices, methods, and computer readable media for improved accuracy of color correction matrix (CCM) coefficient determination based on estimated white point, while maintaining a relatively smooth variation of CCM coefficients over the white point space. The techniques disclosed herein may be achieved via the storage of a limited number of determined CCM multiplier vectors and thus be effective in the camera image pipelines of real image capture devices. With the more accurate CCMs calculated with the disclosed techniques, visible improvement in rendered colors may be achieved as compared to using interpolation from a handful of corner CCMs. The color correction techniques described herein may be implemented by dedicated or general purpose hardware, general application software, or a combination of software and hardware in a computer system.

In this last patent, the added step that Apple is proposing to take is now positioned between "White Point Calculating and Balancing" and "RBG Output."

According to Apple, the camera technology revealed in their four patents can be used in devices such as the iPhone (a mobile phone), the iPod touch (personal data assistant or PDA), an iPod (portable music player), a monitor, a television, MacBook (laptop), iMac (desktop), and iPad (tablet computer(, or other suitable personal device.

Patent Credits

Apple credits Senior Image Scientist Paul Hubel and team members Bai Yingjun, Xuemei Zhang and David Kuo as the inventors of these patent applications which were originally filed under serial numbers 342873, 349090, 348192 and 342879 in Q1 2012.

A Note for Tech Sites Covering our Report: We ask tech sites covering our report to kindly limit the use of our graphics to one image. Thanking you in advance for your cooperation. 

Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.