Correcting Blurry Photo Shots

In respect to cameras, it was a busy week for Apple. In total, they racked up a whopping ten patent filings related to still photography and video coding. In our main report, we cover Apple's patent that is out to diminish blurry photo shots.

The limitations of Embedded Cameras

Many electronic devices today include the ability to capture images. In particular, many cellular telephones, personal assistants, and other portable electronic devices have integrated some or all of the functionality of cameras. Using a lens, the electronic devices could capture light and store images of the user's environment or of people or objects of interest to the user. Because of inherent processing limitations, lens limitations, previewing limitations, and other limitations of lens embedded in electronic devices, users may experience a lag between the time when the users provide an instruction to capture an image and when the device actually captures and stores an image. For example, an electronic device may be subject to an inherent lag as the device switches from a preview mode in which the lens resolution is low (e.g., and requires less power) to an image capture mode having a higher resolution.

During the lag time, a user may shake or unintentionally move or tilt the electronic device, which may cause the image actually captured to be blurry or out of focus. In some cases, the act of pressing a button, touch screen, or other input interface of the electronic device could cause the electronic device, and thus the camera lens, to shake. Typical solutions to the device shake could include image stabilization or post-image capture sharpening and processing. In electronic devices that have smaller sensors however, image stabilization may not be possible. In addition, electronic devices with limited graphics processing capabilities or limited power may not be able to perform post-processing on an image in a resource efficient manner.

Apple's Solution

Apple's patent covers systems and methods for capturing several images in sequence in response to receiving a user instruction to store a captured image – or take a picture. In particular, this is directed to comparing detected movement data associated with each of the several images in the sequence, and selecting the image associated with suitably low movement data.

Apple's patent states that when the user provides the instruction (e.g., presses a button to take a photo), the iPhone (or any iOS device with a camera) could move slightly due to the force applied by the user on the iPhone. In some cases, the iPhone may have a lag between the time the user provides the instruction and the time the iPhone actually records a captured image. During this time, a user may not hold the iPhone perfectly still or immobile (e.g., the iPhone could move relative to the initial position when the user took the picture.

In other cases, the lens may require light to be captured for an extended period during which a user could inadvertently move their iPhone (e.g., in low light conditions). This movement in the device could cause the lens to capture images that are not sharp or that are blurry or out of focus.

To ensure that an image is not blurry, the iPhone could continuously buffer images captured by the lens. The buffer could have any suitable length, including for example a buffer large enough to hold 10, 20 or 50 images. In some embodiments, the buffer length may vary based on user inputs. For example, a buffer may have an initial length of 10 images that expands to 20 images in response to receiving a user instruction to store a captured image. As each image is captured by the lens and stored in the buffer, the motion sensing component could provide an output depicting the device movement at the time the image is captured. Thus, each image in the buffer could be associated with motion information describing the movement of the electronic device at the time the image was captured.

In response to receiving a user instruction to store an image (take a picture) captured by a lens, the iPhone could mark the particular image in the buffer that corresponds to the moment at which the instruction was received. In addition, the electronic device could continue to store captured images in the buffer, up to a predetermined maximum buffer length (e.g., store 10, 15 or 20 additional images). The buffer could then include a series of images captured before and after the instruction was received. To select a picture that will not be blurry, the iPhone could analyze the motion sensing information associated with each of the images in the buffer, and identify the one or more images for which the motion sensing information is less than a maximum threshold or satisfies other criteria. The iPhone could then select one of the one or more images, and store the selected one or more images in memory. In some embodiments, the iPhone could present the one or more images to the user for selection.

Apple credits Bryan James, Andrew Hodge and Aram Lindahl as the inventors of patent application. For more information, see patent number 20100309334, originally filed in Q2 2009.

Skin Tone Mask

In a second camera related patent, Apple introduces us to the problem of traditional color boosting and their related solution which includes utilizing a "skin tone mask."

The Problem: Traditional Color Boosting

One well-known problem with performing a color "boost" operation on a digital image is that it could often have an undesirable affect on areas of the image where skin tones are present. Specifically, skin tones appear to be especially susceptible to boost operations introducing visually unappealing artifacts to the image. For example, certain lighting conditions could give skin tones an over-saturated, unnatural look when boosted. The result is that skin tones appear to have a strong orange, yellow, magenta, or red glow-while the rest of the image appears normally.

Apple's Solution

Conventional image boosting techniques that do not distinguish between skin tone and non-skin tone regions of an image could result in the visually unappealing presentation of human skin tones in certain scenes or light levels. Thus, there is need for a low computational cost, efficient skin tone aware color boosting algorithm for cameras that automatically detects the skin tone and non-skin tone regions of an image and adjusts the color boost in the two regions accordingly to create more visually appealing photos.

One solution that Apple covers in their patent introduces us to the skin tone mask. A "skin tone mask" could be created over the image, covering each sampled value, and classifying each sampled value as either part of the "skin tone mask," e.g., assigning it a value of `1,` or not a part of the skin tone mask, e.g., assigning it a value of `0.` The skin tone mask may then be blurred, e.g., using a Gaussian blur, which may be a Fast Gaussian blur, to account for outliers and holes in the mask and to soften the boundary between the skin tone and non-skin tone regions of the image.

Apple's patent FIG. 6 illustrates a blurred skin tone mask created for a digital image while patent FIG. 7 illustrates an embodiment of a process for carrying out skin tone aware color boosting.

Apple credits Ralph Brunner and Mark Zimmer as the inventors. For more on this, see patent 20100309336, originally filed in Q2 2009.

Radially-Based Chroma Noise Reduction

In a third camera related patent, Apple introduces us to a method and system relating to radially-based Chroma Noise Reduction in the iPhone (or other iOS device with a camera) that involves an edge-preserving blur window.

Apple's patent points us to the phenomenon known as "lens falloff," which concerns the amount of light reaching off-center positions of a sensor. This imaging phenomenon causes the image intensity to decrease radially toward the edges of an image, creating an effect known as "vignetting."

The amount of vignetting present in an image depends upon the geometry of the lens and will vary with various lens properties such as focal length and f-stop. The vignetting effect is more apparent in lenses of lower f-stop (e.g., larger aperture), which are often used in consumer cameras and digital video recorders. Another consequence of the lens falloff effect is that there is often more noise found in pixels farther radially from the center of the sensor or, more precisely, farther radially from the center of the sensors light intensity falloff function, which may or may not be at the geometric center of the sensor.

To correct for this lens falloff phenomenon, a technique known as "flat fielding" could be performed which compensates for the different gains and dark current levels over the surface of the detector.

A part of Apple's solution involves creating a threshold function that would vary radially from the center of the image sensor's light intensity falloff function. A more appropriate threshold value could be chosen for each pixel, allowing for more noise farther from the center of the image, and allowing for less noise closer to the center of the image. Light-product information taken from the image's metadata may be used to scale the threshold value parameters dynamically. This allows the method to perform the appropriate amount of processing depending on the lighting situation of the image that is currently being processed.

Apple's patent FIG. 12 illustrates a graph of radially-dependent expected noise level rings over an image in accordance with one embodiment. FIG. 13 illustrates a perspective view of radially-dependent expected noise level rings over an image

Apple credits Ralph Brunner, Mark Zimmer and David Hayward as the inventors. For more on this, see patent20100309345, originally filed in Q2 2009.

Other Noteworthy Camera Related Patent Applications Published This Week

• Patent Application Number: 20100309344 – Entitled "Chroma Noise Reductions for Cameras
• Patent Application Number: 20100309346 – Entitled "Automatic Tone Mapping for Cameras."
• Patent Application Number: 20100309335 – Entitled "Images Capturing Device having Continuous Image Capture."
• Patent Application Number: 20100309321 – Entitled "Image Capturing Devices using Orientation Detectors to Implement Automatic Exposure Mechanisms."
• Video Related Patent: 20100309985 – Entitled "Video Processing for Masking Coding Artifacts using Dynamic Noise Maps."
• Video Related Patent: 20100309975 – Entitled – "Image Acquisition and Transcoding System."
• Video Related Patent: 20100309987 – Entitled "Image Acquisition and Encoding System."

New Archive Category: Due to this week's barrage of camera patents from Apple, we've decided to add a new Archives category titled "Patents: Camera" for you to check out now or in the future.  

Notice: Patently Apple presents only a brief summary of patents with associated graphic(s) for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for further details. For additional information on any patent reviewed here today, simply feed the individual patent number(s) noted in this report into this search engine. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.

Community Sites Covering our Original Report

Macsurfer, Google Reader, Apple Investor News,  MacTalk (Australia), Hacker News, Techmeme, Fortune (CNN Money), Internet2Go, iPhoneItalia Italy, winandmac, iPhoneHellas Greece, TechBlog (Houston Chronicle), Silicon Valley Blog, MacMagazine Brazil, Gadget Venue, iTrafik France (MacPlus), ZDNet, Mac Life Germany, macnews Germany, The 6sight Report, iPhone Photography, Fone Home UK, Ars Technica, Unwire Hong Kong, United Daily News China. 3C China (MSN), iPhoneography, and more.