The Current Problems Associated with Auto Exposure in Cameras

Today, many personal electronic devices come equipped with digital cameras. Often, these devices perform many functions, and, as a consequence, the digital image sensors included in these devices must often be smaller than sensors in conventional cameras. Further, the camera hardware in these devices often have smaller dynamic ranges and lack sophisticated features sometimes found in larger, professional-style conventional cameras such as manual exposure controls and manual focus. Thus, it is important that digital cameras in personal electronic devices be able to produce the most visually appealing images in a wide variety of lighting and scene situations with limited or no interaction from the user, as well as in the most computationally and cost effective manner possible.

One feature that has been implemented in some digital cameras to compensate for lack of dynamic range and create visually appealing images is known as "auto exposure." Auto exposure (AE) could be defined generally as any algorithm that automatically calculates and/or manipulates certain camera exposure parameters, e.g., exposure time, gain, or f-number, in such a way that the currently exposed scene is captured in a desirable manner.

Auto exposure algorithms are often employed in conjunction with image sensors having small dynamic ranges because the dynamic range of light in a given scene, i.e., from absolute darkness to bright sunlight, is much larger than the range of light that image sensors--such as those often found in personal electronic devices--are capable of capturing.

The difficulties associated with image sensors having small dynamic ranges are further exacerbated by the fact that most image sensors in personal electronic devices are comparatively smaller than those in larger cameras, resulting in a smaller number of photons that could hit any single photosensor of the image sensor.

Thus, there is need for systems, methods, and a computer readable medium for intelligently and dynamically setting a camera's exposure parameters in a visually pleasing way that is independent of the camera's current exposure settings and aware of--and capable of adapting to--the type of scene currently being exposed.

Apple's Solution: Scene Adaptive Auto Exposure

Apple's invention covers systems, methods, and a computer readable medium for an improved automatic exposure algorithm attempt to classify an image into a particular "scene category," and, based on the determined scene category, meter the scene according to a generated metering weighting matrix.

Apple's patent generally describes a new and more effective scene adaptive auto exposure algorithm that compares brightness, i.e., luminance, values in first desired regions of an image to luminance values in second desired regions of the same image, thus remaining independent of the camera's current exposure settings.

In one embodiment, the average luminance is calculated for a central exposure metering region, e.g., a rectangular region centered over the scene, of the image and a pattern of eight peripheral exposure metering regions surrounding the central exposure metering region. Each peripheral exposure metering region could then be assigned a binary value. The 8-bit value corresponding to the binary values of the eight peripheral regions could then be used in sequence as a "scene identifier."

Further, an intelligent auto exposure algorithm as disclosed in Apple's patent, could attempt to categorize the type of scene currently being exposed based on a library of predetermined associations of scene identifiers to "scene categories." In one embodiment, one or more scene identifiers could correspond to a single scene category, e.g., outdoor scene (sunny), outdoor scene (snow at bottom), indoor scene, human subject center, etc. Then, based on the determined scene categorization, the scene could be metered more effectively, e.g., according to one or more predetermined metering weighting matrixes having "target weights" corresponding to the different regions of the scene.

In another embodiment, the metering weighting matrix to be used on the scene may be determined "algorithmically." For example, a target weight may be calculated for each of the peripheral regions of the pattern based on whether the average luminance of the peripheral region is darker, lighter or in the same luminance range as the central region. A target weight may also then be calculated for each of the central regions and adjusted based on the luminance values of the peripheral regions. There may also be different target weights for regions that are not a part of the peripheral regions or the central region. In such an algorithmic embodiment, the target weight value of any region may be further refined based on the extent to which the region's luminance is darker or lighter than the central region.

At one point in Apple's documentation, they state that new scene adaptive auto exposure technique could apply to the iPhone's video camera as well as with the iSight camera in MacBooks and iMacs.

Apple's patent FIG. 4 illustrates a plurality of regions within an outdoor scene with a human subject; FIG. 8 illustrates an adjusted metering weighting matrix for one pattern of exposure metering regions. The first image in the graphic above is from one of the new iPhone 4S ads from Apple. If you look closely at the photo, you'll be able to see a faint sub-grid within the larger nine point master grid being used for cropping the photo. The underlying sub grid is auto adjusting the scene as per this patent's invention.

Apple's patent application was originally filed in Q2 2010 by inventors Frank Doepke and Ralph Brunner.

A Second iOS Camera Related Invention

A second iOS related patent application published this past week was titled "Automatic Tone Mapping Curve Generation Based on Dynamically Stretched Image Histogram Distribution."

Apple's camera-centric invention could only be appreciated by engineers and super camera buffs. Apple's invention covers an apparatus, method, computer useable medium, and processor programmed to automatically generate tone mapping curves in a digital camera based on image metadata.  

Apple states that rather than having a static tone mapping curve for all images, the curve could be varied automatically based on, e.g., the brightness histogram of the image. The improved, automatically generated tone mapping curve may be able to lift shadows more aggressively and/or increase the dynamic range of the image.

About Apple's Patent Figures Noted Above: Apple's patent FIG. 3 illustrates various image color histograms for an outdoor scene; patent FIG. 8 illustrates a tone mapping curve that produces visually-appealing results in a "mid-tone dominated" scene; and patent FIG. 13 illustrates an improved image sensor package for automatic tone mapping.

Apple's patent application 20110292246 was originally filed in Q2 2010 by inventor Ralph Brunner.

iPhone 4S Trivia: Apple's iPhone 4S was used by Matt Garrett of Boston, Massachusetts, to film an 18-20 foot Great White Shark circling the fishing boat he was on during a Thanksgiving get away off the coast of North Carolina. Garrett had only purchased his new iPhone 4 two weeks prior to this incident and was sure glad to have it. Some people have all the fun - don't they, ha!  

Notice: Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. Apple's patent applications have provided the Mac community with a clear heads-up on some of Apple's greatest product trends including the iPod, iPhone, iPad, iOS cameras, LED displays, iCloud services for iTunes and more. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.

Here are a Few Sites Covering this Original Report: MacSurfer, Twitter, Facebook, Apple Investor News, Google Reader, Macnews, iPhone World Canada, MarketWatch, MacDailyNews, iPhoneItalia Italy, iSpazio Italy, KitGuru, TweakTown Australia and US, and more.