On Oct 13, 2011, the US Patent & Trademark Office published a powerful group of seven Apple patents concerning the next round of FaceTime videoconferencing innovations. Some of these innovations include new PIP controls, new focus and exposure operations and a superimposing feature. The depths of these patents are mind numbing as they cover everything from the camera technology and software magic behind FaceTime through to actual user examples. With Microsoft likely to integrate Skype into Windows 8 in 2012 or thereafter, Apple knows that they have to advance FaceTime with important conferencing features for the consumer and the enterprise user in order to stay ahead of the pack.
Apple's Patent Background
Many of today's portable devices, such as smartphones, provide us with video capture functionality. A user of the portable device could capture both still images and video through a camera on the phone. However, to transmit captured video to another party, the user must generally either send the video directly to the other party or upload the video to another location (e.g., an Internet video hosting site) after the video is done being captured. Unfortunately, this doesn't allow the other party to view the live video stream as it is captured by the portable device.
In addition, standard portable devices are only equipped with one camera, and processing information from this one camera is difficult enough. An ideal device would have multiple cameras and could send out live video that is a composition of video from at least two cameras. This is an especially difficult problem in light of the limited resources available for portable devices, both in terms of the device processing multiple captured video streams and a network to which the device is connected handling the transmission of the live video streams.
An Overview of Apple's Proposed Solution
Apple's invention covers an iPhone with dual cameras and a network interface to transmit captured images to one or more devices during a real-time FaceTime session between the users of the devices. The iPhone will also include an encoder that could be used to encode the captured images for local storage or for transmission to another device. The iPhone will further include a decoder that allows the device to decode images captured by another device during a real-time communication session, or FaceTime session, or to decode images stored locally.
One example of a real-time communication session that involves the transmission of the captured video images is a video conference. In some embodiments, the mobile device could only transmit one camera's captured video images at any given time during a video conference. In other embodiments, however, the mobile device could transmit captured video images from both of its cameras simultaneously during a video conference or other real-time communication session.
During a video conference with another device, the mobile device of some embodiments could transmit other types of content along with the video captured by one or both of its cameras. One example of such other content includes low or high resolution picture images that are captured by one of the device's cameras, while the device's other camera is capturing a video that is used in the video conference.
Other examples of such other content include (1) files and other content stored on the device, (2) the screen display of the device (i.e., the content that is displayed on the device's screen), (3) content received from another device during a video conference or other real-time communication session, etc.
Novel In-Conference Adjustment Techniques
The mobile devices of some embodiments employ novel in-conference adjustment techniques for making adjustments during a video conference. For instance, while transmitting only one camera's captured video during a video conference, the mobile device of some embodiments could dynamically switch to transmitting a video captured by its other camera. In such situations, the mobile device of some embodiments notifies any other device participating in the video conference of this switch so that this other device could provide a smooth transition on its end between the videos captured by the two cameras.
In some embodiments, the request to switch cameras not only could originate on the "local" device that switches between its cameras during the video conference, but also could originate from the other "remote" device that is receiving the video captured by the local device. Moreover, allowing one device to direct another device to switch cameras is just one example of a remote control capability of the devices of some embodiments.
Examples of other operations that could be directed to a device remotely in some embodiments include exposure adjustment operations (e.g., auto-exposure), focus adjustment operations (e.g., auto-focus), etc. Another example of a novel in-conference adjustment that could be specified locally or remotely is the identification of a region of interest (ROI) in a captured video, and the use of this ROI identification to modify the behavior of the capturing camera, to modify the image processing operation of the device with the capturing camera, or to modify the encoding operation of the device with the capturing camera.
Yet another example of a novel in-conference adjustment of some embodiments involves real-time modifications of composite video displays that are generated by the devices. Specifically, in some embodiments, the mobile devices generate composite displays that simultaneously display multiple videos captured by multiple cameras of one or more devices. In some cases, the composite displays place the videos in adjacent display areas (e.g., in adjacent windows). In other cases, the composite display is a picture-in-picture (PIP) display that includes at least two display areas that show two different videos where one of the display areas is a background main display area and the other is a foreground inset display area that overlaps the background main display area.
The real-time modifications of the composite video displays in some embodiments involve moving one or more of the display areas within a composite display in response to a user's selection and movement of the display areas. Some embodiments also rotate the composite display during a video conference, when the screen of the device that provides this composite display rotates. Also, the mobile device of some embodiments allows the user of the device to swap the videos in a PIP display (i.e., to make the video in the foreground inset display appear in the background main display while making the video in the background main display appear in the foreground inset display).
Random Assortment of Patent Graphics
Apple's patent FIG. 23 noted above illustrates a user interface of some embodiments for a focus adjustment operation; FIG. 32 illustrates a user interface of some embodiments for a snap-to-corner operation; FIG. 38 illustrates a user interface of some embodiments for resizing a foreground inset display area in a PIP display.
In patent FIG. 41 shown below we see that you'll be able to pinch-and-zoom the PIP display: In FIG. 52, we see a new remote switch camera button 5288 is shown in a display area 1155 along with a mute button 5282, an end conference button 5284, and a local switch camera button 5286.
Apple's patent FIG. 54 shown below illustrates another user interface of some embodiments for a remote control switch camera operation; FIG. 63 illustrates another user interface of some embodiments for a focus and/or exposure adjustment operations.
Some embodiments allow a dual camera mobile device to display videos captured from the mobile device and videos captured from another dual camera mobile device during a video conference in any of several display arrangements. FIG. 65 shown below illustrates examples of different display arrangements for videos captured from one or more dual camera mobile devices. In FIG. 65, a user of a dual camera mobile device 6500 (Device A) and a second user of a second dual camera mobile device 6505 (Device B) are having a video conference with each other.
Some embodiments allow a user of a dual camera mobile device to superimpose a foreground of a video onto another video in a PIP display during a video conference. In some embodiments, the foreground of a video blends into the other video in such a way that they appear as if a display of a single video captured by a single camera. Apple's patent FIG. 66 shown below illustrates an example of such superimposing of a foreground of an inset video onto a background video in a PIP display.
Apple's patent application was originally filed in Q2 2010 by inventors Stephen Lemay – who's been behind numerous Apple patents, Elizabeth Cranfill, Joe Abuan, His-Jung Wu, Xiaosong Zhou and Roberto Garcia Jr. Apple's powerful patent is one within a group of seven deeply detailed patents.
Notice: Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. Apple's patent applications have provided the Mac community with a clear heads-up on some of Apple's greatest product trends including the iPod, iPhone, iPad, iOS cameras, LED displays, iCloud services for iTunes and more. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.
Here are a Few Great Community Sites covering our Original Report
MacSurfer, Twitter, Facebook, Apple Investor News, Google Reader, UpgradeOSX, TechWatching, Macnews, iPhone World Canada, CBS MarketWatch, MacDailyNews, Techmeme, MacPrime Switzerland, and more.
