Apple's Patent Background

As screen real estate becomes increasingly valuable, systems have been developed to use a secondary display screen to either mirror, or provide additional display area to, a primary display screen of a computing device. In the mirroring scenario, what is displayed on a primary display screen of a computing device (e.g., a laptop) may be "mirrored" by a secondary display screen (e.g., attached monitor) that is, for instance, larger and capable of higher-resolution display. In the extension scenario, the secondary display simply provides additional screen real estate but does not "mirror" the content of the primary display. For examples, a user may use the primary display to view one website and use the secondary display to view another website.

However, conventional approaches for using a primary display with a secondary display have shortcomings. First, users have to manually configure the settings of the primary and secondary displays, a process that can confuse and intimidate a significant number of users. Second, even for users who are capable of performing such configuration, the process is time-consuming, inefficient, and irritating to the users' overall computing experience. Third, even technically savvy users may fail to discover display arrangements and/or settings that would maximize the advantage of having two displays.

Apple Invents Interactive AirPlay

Apple's invention generally relates to presenting information in an electronic environment and in particular to content presentation and manipulation using multiple electronic devices. In a simpler way of viewing this, it's advancing today's AirPlay so that it's an interactive experience. One that could also allow different interfaces to work with each other.

For instance, an iPad could become a game controller for a video game on your TV as noted below in patent FIG. 8.

Apple also describes an alternative application such as a DJ application or an application that allows users to create music can also cause the controls to be displayed on the touchscreen device while album art or a music score is displayed on the display device. For instance, the DJ application can display a set of controls including a selectable surface of a drum, piano keyboard, etc. to be displayed on the touchscreen device. The user may then create music or cause album art or visualizer to be displayed on the display device by performing a series of user interaction with the set of controls.

Apple's graphic shows Xbox gaming logo and this week Microsoft announced that the were bringing Xbox live to iOS.

Apple further notes that certain embodiments of the present invention relate to content presentation across multiple electronic devices. In some embodiments, content of a software application can be presented across the displays of multiple electronic devices in an intuitive and coordinated fashion. Below is patent FIG. 11 illustrating Apple's Keynote app on the iPad with just the current slide showing up on a television of other kind of secondary display that non-touch based.

User interfaces displayed by the displays of the electronic devices may represent different aspects or different levels of a set of information items in the software application.

In another application, Apple notes that in response to receiving a user input via a user interface of an electronic device, another user interface of another electronic device may be modified based on the user input. For example, a tablet computer may display a user interface including a list of songs for a media player application and a desktop monitor may display another, distinct user interface including detailed information for a particular song from the list, in response to a user selection of the particular song via the user interface. Below is patent FIG.

Certain embodiments of the present invention enable coordinated interaction and presentation across a touch device and a non-touch device. A touch device (also referred to as touchscreen device) can be any electronic device that can receive user input via a touchscreen or a display of the device, such as a tablet computer or a smartphone.

A non-touch device (also referred to as display device) can be any electronic device that cannot receive user input directly through a display coupled to the device, such as a conventional light emitting diode (LED) computer monitor or a liquid crystal display (LCD) television screen.

Two Sets of Information on Differing Devices

In some embodiments, a touch device can establish a connection with a non-touch device. At least one of the devices can execute an application where content of the application may be presented across the displays of the devices. A set of user-selectable user interface items can be displayed on the touch-enabled display of the touch device. The electronic device can receive a user selection of one of the set of user interface items displayed on the touch-enabled display. In response to receiving the user selection, another set of information items can be presented on the non-touch device.

Other Apple Examples of Interactive Airplay

FaceTime Application Example

In Apple's Patent FIG. 10 shown below we're able to see another example of interactive AirPlay which shows how different levels of information that can be displayed on a TV and an iPad. In this example, FaceTime is running on a TV. The application generates a first user interface including a video image of the local user for display on an iPad. The application further generates a second user interface including a video image of another participant (or multiple other participants) with whom the user is having the conference for display on a TV.

By presenting an arrangement of the video images that would be intuitive to a user (e.g., where the user's local video is shown on an iPad and the remote participant's feed is viewable through a TV), the application can provide the user a better user experience in using the application.

In some embodiments, a number of control elements such as menus, slider bars, text fields, and buttons representing volume adjustments, lighting adjustments, focus adjustments, and/or zoom adjustments functionality can be displayed on the iPad such that the user may make adjustments when desired by performing a user input through the touch-enabled display of the iPad. As such, the remote video need not be obstructed by control elements. The controls may also fade out in some embodiments when the user has not interacted with the touch-enabled display beyond a threshold period of time (e.g., after 5 seconds), thereby increasing the screen real estate for the user's video image.

AirPlay: Interactive Maps

In Apple's patent FIG. 9 shown below we see yet another example of AirPlay in Interactive Mode. Apple's patent FIG. 9 illustrates another example of how different levels of information can be displayed on a display device and a touchscreen device. In this example, Apple's Maps application is being executed on a TV and an iPad. Maps will be able to generate views of different perspectives and for different levels of detail of a particular map area on the iPad and TV in response to a user command.

As shown in this example, the iPad shows a bird's eye view of a map area while the TV shows a street level perspective view of a particular location within the map area.

In some embodiments, the iPad may further display a set of selectable controls that the user can select to indicate the direction of the street view to display on their TV.

Additional levels of mapping information can be provided by the user interface displayed on the TV as well, including a list of stores, restaurants, and gas stations around a particular area indicated by the user (e.g., via a selection of a particular location on the map on the user interface displayed the iPad) or other types of information that can be provided about the particular location.

Patent Credits

Apple credits Julian Missig, Jeffrey Bernstein and Marcelo Coelho as the inventors of patent application 20140068520 which was originally filed in Q3 2012.

A Note for Tech Sites covering our Report: We ask tech sites covering our report to kindly limit the use of our graphics to one image. Thanking you in advance for your cooperation. 

Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments.