Overview of the Problem

Video stabilization is a class of video processing that removes unwanted shakiness from videos captured from portable camera devices such as smart phones, personal entertainment systems laptop computers and/or camcorder. The goal of video stabilization is to revise an original video sequence to mimic a sequence that would have been obtained if a camera captured the video from an ideal or a specified motion trajectory.

Specifically, video stabilization techniques generate an idealized motion vector of a captured video sequence and then introduce motion compensation to a sequence of captured video to replicate the idealized motion vector. If, for example, a video stabilization algorithm estimated that a video sequence should exhibit no motion (e.g., ideally a camera would have been perfectly still during motion capture), then the motion compensation processes would estimate a global motion vector on each frame and perform processes to remove the global motion.

Although video stabilization could improve the perceptual quality of a video sequence, it has its consequences. First, it could consume considerable resources at a capture device or processing device. Second, it could reduce the field of view of the final video sequence. Third and perhaps most importantly, video stabilization could impair perceived quality if the algorithm generates an incorrect estimate of idealized motion or an incorrect estimate of the source motion vector.

Increasingly, consumer capture devices are provisioned with motion detection devices such as accelerometers and/or gyroscopes. The motion detection devices could provide metadata that indicates motion effects of a camera during video capture, however, even though the motion detectors provide data relating to global motion of the camera, the level of shakiness between frames often is comparable to the noise level of the motion detector data. Such high level of the noise in data prohibits direct use of accelerometer data in video stabilization.

Apple's Solution

Apple's invention provides a control system for video processes that selectively controls the operation of motion stabilization processes. The method includes capturing video via a camera to generate a captured video sequence, aggregating motion sample data from a motion detection device over the period of each frame in the captured video sequence, comparing the aggregated motion data to a threshold, and identifying, based on the comparison, portions of the captured video sequence that represent a scene change, and performing video stabilization on portions of the captured video sequence that are outside the scene change.

A Look at the Proposed Video Stabilization System

Apple's patent FIG. 1 is a simplified block diagram of a portable video device 100 that could be MacBook an iOS device, or other portable device such a third party video camera. The video device may include a processing system, a camera, and a motion detector such as an accelerometer or gyroscope. The camera may include a lens system and imaging device to convert incident light into a digital data stream. The motion detector may generate electrical data indicating a direction of movement of the mobile device.

An Overview of the Various Stabilization Stages

Apple's patent FIG. 2 as noted above illustrates a process flow diagram illustrating data flow according to an embodiment of the present invention. In this embodiment, captured video data and motion data may be processed by a global motion estimation stage, a motion detector processing stage, a scene change detection stage, a motion smoothing stage and a motion stabilization stage.

In the global motion estimation stage, a video device may calculate motion of video content on a frame-by-frame basis across a field of view. The global motion estimation stage may output metadata identifying, for each frame in the captured video, a motion vector representing average motion of the frame, measured from a preceding frame. The motion estimate metadata may be output to the motion smoothing stage and the scene change detection stage.

The motion smoothing stage may generate new motion vectors for each frame according to average motion observable in the motion vectors output from the global motion estimation stage. Motion smoothing helps remove jitter and other high frequency artifacts from the motion vectors output by the global motion estimation stage. The motion smoothing stage may output motion vectors to the motion stabilization stage.

The motion detector processing stage may receive motion data from a motion detector device. The motion detector outputs motion data at rate in excess of one sample per video frame.

The scene change detector stage may selectively enable and disable operation of the motion stabilization stage based on motion values provided by the motion detector processing stage.

The Video Coder

In Apple's patent FIG. 6 illustrates a video coder that includes a video stabilization unit and a motion smoothing unit. The motion smoothing unit may receive motion samples from the motion detector or, optionally, from the motion detector processor. The motion smoothing unit may output revised motion vectors to the video stabilization unit for use in stabilization processing.

Software and Hardware Solutions

The techniques described in Apple's patent pending invention find application in both software- and hardware-based control systems. In a software-based control system, the functional units described may be implemented on a computer system (commonly, a server, personal computer or mobile computing platform) executing program instructions corresponding to the functional blocks and methods listed in the patent. Hopefully this will translate to being available on iMovie.

The program instructions themselves may be stored in a storage device, such as an electrical, optical or magnetic storage medium, and executed by a processor of the computer system. In a hardware-based system, the functional blocks illustrated above may be provided in dedicated functional units of processing hardware, for example, digital signal processors, application specific integrated circuits, field programmable logic arrays and the like. The processing hardware may include state machines that perform the methods described in the foregoing discussion. The principles of the present invention also find application in hybrid systems of mixed hardware and software designs.

Apple's patent application 20110234825 was originally filed in Q2 2010 by inventors Yuxin Liu, Xiaojin Shi, James Normile and His-Jung Wu. 

One Last Tiny Patent Pending Innovation to Point Out

I just wanted to point out how something as small as an iOS device earphone Jack gets the royal treatment for design scrutiny in Apple's Labs as does the iPhone. The level of detail outlined in Apple's patent graphics is stunning really. This is patent application 20110237131 titled Audion Connector having Additional Detection Switch. 

   

Apple's Abstract: Circuits, methods, and apparatus for improved audio connectors. One example provides an audio connector, that, for purposes of activating circuitry to receive signals from a microphone, does not detect the presence of an audio jack until the audio jack is inserted far enough into the audio connector that a microphone contact on the audio jack comes into contact with a microphone signal pin on the audio connector. To prevent the redirection of audio signals away from an audio jack when the audio jack is partially extracted from the audio connector, such partial extractions are not detected by audio signal pins.

Notice: Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. Apple's patent applications have provided the Mac community with a clear heads-up on some of Apple's greatest product trends including the iPod, iPhone, iPad, iOS cameras, LED displays, iCloud services for iTunes and more. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.

Here are a Few Great Community Sites covering our Original Report

MacSurfer, Twitter, Facebook, Apple Investor News, Google Reader, UpgradeOSX, TechWatching, Macnews, iPhone World Canada, CBS MarketWatch, MacDailyNews, Applesfera Spain, Sebastian Sweden, Tecnetico Spain, MacNN, iPhone-Ticker Germany, CNET, and more.