One of the forty patent applications that were published by the US Patent & Trademark Office for Apple this past week focused on advancing FaceTime conferencing. Years ago, the vast majority of video conferences were conducted in controlled studios or conference rooms designed with specialty lighting for optimal viewing. Today that's all changed; the vast majority of video conferences are done on the fly, on the go, in a meeting, in a park, in your car, in a lobby or in your home office where lighting conditions could be insufficient or constantly changing. The standard Auto Exposure (or AE) features found in most mobile phones today weren't designed to adapt to those kinds of quickly changing lighting environments for mobile video conferences. Apple's invention illustrates the intent is to go far beyond the auto exposure standards of today to deliver a solid solution to this problem.
Today's Problem with Auto Exposure for Video Conferencing
Today, many personal electronic devices come equipped with digital video cameras. Often, the cameras in these devices perform many functions, such as: image capture, video capture, and videoconferencing. Videoconferencing with handheld and/or mobile personal electronic devices is a much more difficult endeavor than traditional videoconferencing applications, wherein the camera is mostly fixed and the lighting and/or composition of the scenery around the videoconferencing participant(s) doesn't often change drastically. Thus, it is important that digital video cameras in handheld and/or mobile personal electronic devices be able to capture visually appealing images in a wide variety of lighting and scene conditions with limited or no interaction from the user, while at the same time allowing for the video encoding of the captured image frames to be carried out in the most computationally effective and visually appealing manner.
One feature that has been implemented in some digital cameras to compensate for the lack of dynamic range of the relatively small image sensors typically found in handheld and/or mobile personal electronic devices is known as "auto exposure." Auto Exposure (AE) could be defined generally as any algorithm that automatically calculates and/or manipulates certain camera exposure parameter values, e.g., exposure time, gain, or f-number, in such a way that the currently exposed scene is captured in a desired manner.
In particular, AE methods that are tuned for fixed camera videoconferencing applications typically "meter" or calculate the brightness on a relatively large area of the image with a reasonably tight "AE stability region" centered around a fixed, target brightness value (an "AE Target"). In such fixed camera videoconferencing applications, the target brightness value may often be set at or above 18% of the maximum capturable signal strength of the image sensor.
The relatively high target brightness values and relatively tight stability regions typically employed in fixed camera videoconferencing applications could be problematic for handheld and/or mobile videoconferencing applications, in which the scene can be strongly back lit or front lit, and in which the scene lighting levels can change often and dramatically.
In mobile videoconferencing applications, it is also paramount to ensure the highest possible video quality at the lowest possible bit rate. Existing video codec standards today, e.g., H.263 or H.264, employ both inter-coding and intra-coding techniques. Intra-coding techniques are performed relative to information that is contained only within the current video frame and not relative to any other frame in the video sequence. Inter-coding techniques, on the other hand, involve temporal processing, that is, rather than resending all the information for a subsequent video frame, the codec will only encode and send the changes in pixel location and pixel values from one video frame to the next video frame, while making "temporal predictions" to attempt to take advantage of temporal redundancy between subsequent image frames, thus allowing for higher compression rates. However, the more gradually an AE method changes exposure parameter values, the longer the image frame's average brightness levels are changing, thus making it more difficult for the encoder to make accurate temporal predictions for inter-coding.
Thus, there is need for systems, methods, and a computer readable medium to intelligently and dynamically set a digital video camera's exposure parameters in such a way as to lead to visually pleasing images and efficiently encoded video streams in handheld and/or mobile videoconferencing applications.
Apple's FaceTime to Gain New Techniques to Improve Auto Exposure
Apple's solution overview begins by stating that auto exposure (AE) algorithms in handheld and/or mobile videoconferencing applications--unlike their fixed camera counterparts--are often exposed to a wide variety of rapidly changing lighting and scene conditions, and thus face a difficult trade-off between adjusting exposure parameter values too frequently or not frequently enough.
In personal electronic devices executing such handheld and/or mobile videoconferencing applications, it may be desirable to adjust exposure parameter values gradually, so as to reduce video oscillations causing visually jarring effects on the device's preview screen due to rapidly changing brightness levels. However, adjusting the exposure parameter values too gradually could cause the video encoder to make inaccurate temporal predictions, resulting in poor quality video streams over a large number of frames. Thus, Apple has discovered various techniques to improve auto exposure methods for use in variable lighting conditions, especially as applied to handheld and/or mobile videoconferencing applications executing on personal electronic devices.
FaceTime will implement a Center-Weighted Metering Weighting Matrix
In one of the two embodiments noted below in Apple's patent FIGS. 4 and 5, we see a small, centered exposure metering region or "an exposure metering rectangle" that is used to meter on the subject of the video capture. With handheld videoconferencing applications, the user is generally able to keep the camera pointed at the subject of interest, such that the subject of interest is almost always centered in the image frame. By utilizing a small, centered exposure metering region with a center-weighted metering weighting matrix, this embodiment takes advantage of the likely-centered nature of the subject of interest in the image frame.
Apple's patent FIG. 6 shown below relates to FIG. 5 above. In patent FIG. 6 we see an iPhone with a mobile videoconferencing application (future version of FaceTime) being executed on the iPhone's preview screen 210. By utilizing preview window (604) the user (102) may ensure that his face remains centered in the image frame due to setting up the small, centered and center-weighted exposure metering region, such as that described in relation to FIG. 5 above. Once set up it will continue to produce satisfactory auto exposed image frames.
Apple's patent FIG. 8 noted above illustrates an in flowchart form, one embodiment of a process for auto exposure in a handheld/mobile personal electronic device using an AE stability region.
FaceTime to Gain a New Auto Exposure "Stability Region"
In another embodiment, Apple introduces us to the "AE stability region," that will be used to aid in the Auto Exposure's methodology of determining when (and to what degree) to adjust exposure parameter values.
The AE stability region defines a range of acceptable brightness values for the image frame. The AE stability region may be centered on a fixed, target brightness value. If the "AE Average" for the image frame, i.e., the calculated weighted brightness average over the desired exposure metering region(s), is within the AE stability region, the embodiment may leave the camera's exposure parameter values at their current levels. However, if the "AE Average" for the image frame is outside of the AE stability region (either higher than the upper limit of the AE stability region or lower than the lower limit of the AE stability region), the embodiment may change the camera's exposure parameter values appropriately to adjust the next frame's AE Average back towards the stability region by a predetermined step size.
The Auto Exposure Techniques for Variable Lighting Conditions patent finishes with discussing "AE convergence Speed." A fast AE convergence speed, along with a large AE stability region, could improve the appearance of a scene lighting transition by quickly reaching a new "stable" AE Average point. Using a fast AE convergence speed may also aid in the efficiency of the video encoding in videoconferencing applications.
Although Apple's auto exposure patent focused on mobile handsets, Apple clarifies that it will also apply to other products such as the MacBook, iMac and Mac Pro with their new Thunderbolt Display that includes an HD camera for FaceTime use.
Apple's patent application was originally filed in Q3 2010 by inventors Douglas Price, Xiaosong Zhou and His-Jung Wu. Our cover graphic uses Apple's patent figure 2 which is illustrating a user that's outdoors conducting a FaceTime conference.
The Use of Our Graphics: We'd ask those covering our report to limit the use of our graphics to a single graphic. Thank you.
Notice: Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. Apple's patent applications have provided the Mac community with a clear heads-up on some of Apple's greatest product trends including the iPod, iPhone, iPad, iOS cameras, LED displays, iCloud services for iTunes and more. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.
Here are a Few Sites covering this Original Report: MacSurfer, Twitter, Facebook, Apple Investor News, Google Reader, Macnews, iPhone World Canada, MarketWatch, MacDailyNews and more.