Apple's Patent Background

 

Several connected devices can share information in many different contexts. For example, several devices can share contact information, communications (e.g., e-mail messages), application data, or other information to be used independently by each of the several devices. As another example, several devices can share common data used by a same application or process operating on each of the devices. In particular, several users can play a game together by sharing game information between the electronic devices.

 

Some processes or applications operating cooperatively between several devices may require different devices to provide information in a specific order. For example, some games (e.g., card games, battleship, or other board games) may be played by allowing each user, in a specific order, to play cards from the user's hand. Different approaches can be used to ensure that devices provide information in a correct order. For example, the users of the devices can coordinate amongst themselves to provide information at a proper time.

 

As another example, users can assign an order or sequence to each of the devices, such that devices only provide information when the sequence allows them to do so. These approaches, however, can create an additional burden on the users.

 

In another context, users having electronic devices may be in a location where positioning circuitry is unavailable (e.g., a large conference room or a concert hall), and may wish to locate each other using the devices. To do so, the users may need to send various messages to each other (e.g., by e-mail, text message, or telephone calls) describing where they are, what they see, and try to guess where other users are to establish a suitable meeting point. This process can be time consuming and inaccurate. In addition, in noisy environments, voice communications between users can be further impeded. Apple's invention is to provide solutions to these problems.

 

Apple's Solution Overview

 

Apple's invention is directed to determining the relative position of several devices based on images of a tag captured by each of the several devices. In particular, the invention is directed to monitoring images of a tag captured by different devices, sharing the captured images, and determining the relative positions of each device based on the shared images.

 

Creating a Tag via a Photo of a Common Object in your Environment

 

Apple states that to determine the relative position of several devices, each device can capture an image of a same tag via their device's built-in camera. Any suitable object can serve as a tag of which an image is captured. In particular, any object for which two images captured from different perspectives are always different can be used as a tag (e.g., an asymmetrical object).

 

In some cases, users can instead or in addition capture an image of an object in the users' environment, or of a region of the users' environment. In such cases, any object in a user's vicinity can serve as a tag.

 

Orientation of the Device Relative to a Tag

 

According to Apple, a tag can include an identifier or other reference indicating to a user one or more properties of the tag. The reference can include information describing features of the tag relative to a coordinate system. Alternatively, the electronic device can contact a remote tag server or other information source that relates to tags to retrieve information corresponding to the tag (e.g., retrieve a copy of the tag). When the tag is known, the electronic device can compare the known tag with the captured image of the tag, and determine the orientation of the device relative to the tag. For example, the electronic device can determine an angular measurement depicting an orientation of the device relative to a portion of the tag (e.g., a corner of the tag serves as an origin for a reference system).

 

As another example, the electronic device can define a position, an orientation, or both in a two or three-dimensional reference system relative to the tag. For example, the electronic device can define a matrix providing positioning information for the device relative to the tag.

 

Once the electronic device has determined its position or orientation relative to the tag, the electronic device can provide the determined position or orientation information to other electronic devices positioned around the tag. For example, the electronic device can transmit a positioning matrix to other devices.

 

In another example, Apple states that the electronic device can instead or in addition receive positioning information from the other devices, and determine the relative positions of the devices (e.g., which device is to my left, to my right). In some embodiments, the electronic device can define a map on which all of the devices are positioned to allow a user to view the distribution of devices, confirm that the device has detected the correct distribution of other devices, or combinations of these.

 

Example: Finding a Friend at a Concert Hall

 

Apple states that in some embodiments, a location or site visited by users can provide one or more tags visible from most or all of locations within the site such that users can capture images of the tags and determine their positions relative to the tags from anywhere on site. For example, a concert location can display a single tag above the stage as patent FIGS. 8 and 9 illustrate below.

 

Apple's patent FIG. 7 above is schematic view of an illustrative display of an application for finding a path to another electronic device. Apple is extending their Find My iPhone with "Find My Friend." In some embodiments, the tag can be provided by a particular site (e.g., by a concert hall or convention center).

 

In Apple's patent FIG. 8 noted above we see a schematic view of an illustrative display for indicating the positions of several devices. In the example of a concert hall, the map can include seating sections, a stage, and representations of several levels (e.g., an orchestra level and a balcony level). In some embodiments, a user can navigate within the map, for example by panning or scrolling the map, zooming in and out different regions of the map, or selecting displayed items in the map to view more detail or additional information.

 

The devices of the user and the user's friends can be identified by individual markers. For example, marker 820 can identify the user's device, while markers 822 and 824 (noted in light blue) can identify each friend's device.

 

Using the marked locations, a user can determine an appropriate path to go to the location of one of the user's friends. For example, a user can identify several landmarks around which a user will need to move to reach a friend. As another example, the user can direct the electronic device to define a path that the user can follow to reach his friend.

 

In some embodiments, the user can select a particular marker to provide a communication to the corresponding friend, or to direct an application of the user's device to interact with an application of the friend's device. For example, a user can select a marker to provide a communication directing a friend to meet the user at a particular location. As another example, a user can select a marker to direct a friend to open an application used to share recorded information (e.g., share a video of concert goers dancing to music, or a video of a band on stage).

 

In some cases, a user can instead or in addition send a request to a friend to come to the user, or to meet the user at a meeting point. For example, the user can send a communication to another user inviting the user to come to his location (e.g., send a message indicating why the user's current location is better than the friend's location, and inviting the friend to join the user).

 

Apple's patent FIG. 9 is a schematic view of an illustrative display for prompting a user to capture a new representation of a tag. The electronic device can prompt a user to capture a new representation of a tag by displaying prompt 940, for example as a pop-up window. In some cases, a user may need to re-orient the device to align a sensor with the tag (e.g., point a camera of the device towards the tag).

 

The electronic device can direct a user to capture new representations of the tag at any suitable interval. In some embodiments, the electronic device can define a time-based interval, and prompt a user to capture a revised tag representation when the interval has lapsed (e.g., every 3, 5 or 10 minutes). In some embodiments, the electronic device can instead or in addition monitor the movement of the device to predict or evaluate how far a user has moved since a representation of the tag was last captured.

 

In other cases, Apple notes that a tag can be provided as a three-dimensional object, such that the tag can be captured from any perspective (e.g., a cube or a pyramid). The particular portion of the tag that is captured can include embedded information describing the orientation of the captured portion or face of the tag. This will play right into the strengths of Apple's newly acquired high-end 3D mapping company.

 

Tags can include Embedded Bar Codes, GPS Coordinates and/or a Map

 

In some cases, Apple states that a tag can include a code (e.g., an embedded bar code) that can be used to identify the particular tag or provide attributes of the particular tab (e.g., characteristic dimensions or GPS coordinates).  

 

In some embodiments, each electronic device can indicate to its user the relative position of other devices having captured images of the same tag. For example, an electronic device can display a map or other representation of the user's environment in which the other detected devices are placed. The device can provide distance and angle information for each other device including, for example, a vector or directions for reaching the other device. In some embodiments, the directions can account for known features of the environment (e.g., walls and doors, roads, or other obstacles).

 

The use of GPS could be used in context with maps and movement related to a vehicle in motion: think turn by turn instructions.

 

Multi-Player Games

 

A second aspect of Apple's invention revolves-around Multi-Player gaming. Here, electronic devices can determine their relative positions at any suitable time. In some cases, electronic devices can monitor the device's perspective of a tag and provide changes in the perspective or position to other devices. Once the relative position of each device has been determined, the devices can share information or content used by applications operating on the devices.

 

For example, several devices can share content related to a multi-player game application operating on each device. The determined positioning information can be used to control the game play, and the order in which different users provide content to other devices.

 

Users of each device can interact with displayed content using different approaches. For example, a user can provide an input on a device to displace or move the content. To ensure that all of the other devices see the interaction of the user with the content, the electronic device of the user can broadcast the user's interactions to the other devices. In some cases, several users can simultaneously or sequentially interact with displayed content, for example as part of a game. For example, several users can interact with one or more displayed avatars that fight each other in a game.

 

In some embodiments, several electronic devices can engage in a cooperative application where the relative position of each device is used as a factor or input in the cooperative application. For example, several electronic devices can display a virtual avatar or other virtual content, where each device displays the avatar from a different perspective based on the position of the device relative to a tag. As a device moves, the displayed perspective of the avatar can change.

 

Multi-Player Game Example

 

In Apple's patent FIGS. 14A and 14B shown below we see schematic views of illustrative displays of virtual content with which several users can interact. Display 1400, shown in FIG. 14A, can include tower 1410 of game elements. For example, the tower can be constructed by stacking different blocks or bricks. A user can play the game by removing a brick from the tower and replacing the brick on the top of the tower. If the user causes the tower to topple when the user removes the brick, then the user loses. If the user successfully moves a brick, the following user (e.g., using a different device) can interact with the tower. Several users can interact with the same tower using different devices.

 

Several devices used by different users can display the tower and game elements so that users of the different devices can play the game together. In some cases, the particular portion or perspective of the displayed tower and game elements can correspond to a perspective of a device relative to the tag. Display 1420, shown in FIG. 14B, can include tower 1430 of game elements (including game element 1432), which can correspond to tower 1410 and game element 1410. In particular, the virtual content provided on display 1420 can be the same as the virtual content displayed in display 1400, but displayed from a different perspective.

 

Using a Mac or iPad to Host & Provide Coordinated Actions & Perspectives for Multi-Player Games

 

Apple states that in some instances, a host device can serve as a game provider, and can provide information to each device describing the commonly displayed content. For example, a larger portable device (e.g., a notebook or tablet computer), or a fixed device (e.g., a desktop computer or a server) can generate content that is transmitted to the different devices.

 

When different devices interact with the content, the devices can transmit their interactions or received user instructions to the host device, which can in turn adjust the manner in which the content is displayed, and provide the adjustments to the different devices. In some embodiments, each device interacting with the displayed content can instead or in addition provide the interactions or instructions directly to other devices displaying the content.

 

The host device can provide the content to be displayed by each device using any suitable approach. In some embodiments, the host device can determine the particular view of the content that each device will display, and provide individualized perspectives of the content to each device. When a particular device interacts with the content, the device can provide information describing the interaction to the host device so that the host device can send revised perspectives of the content to the users. This is exemplified in patent FIG. 10 below in our Card Game Example.

 

Multi-Player Poker Game

 

To help you better understand the role of a host within a multi-player gaming environment, we look to patent FIG. 10 below. In this particular example, the shared information can include several cards usable by any player in a poker game.

 

In many card games, cards can be distributed by a dealer in a particular order. Users can then play in the same order in which cards were dealt. In the example of FIG. 10, devices 1010, 1020, 1030 and 1040 can determine their relative positions such that the applications operating on each device can coordinate to ensure that each device receives cards, or plays cards, in an appropriate order.

 

For example, cards can be dealt in clockwise or counterclockwise order with a new dealer each round. Users can place their bets and change cards (if allowable), in the dealing order. The applications can coordinate to only allow the proper user to play by preventing other users (e.g., users other than the one whose turn it is to play) for providing inputs corresponding to the game play (e.g., place a bet, fold, request cards, etc.). When a user whose turn it is to play provides an appropriate instruction to the application, the application of the user's device can share the user's play with the other devices, and allow the device associated with the next user to play.

 

The Game's Definition File

 

Alternatively, the host device can provide definition information defining the content from some or all perspectives or sides. For example, the host device can provide definition information for all perspectives, or for perspectives corresponding to a particular range of device positions of the initial position of the device relative to the content.

 

When a device changes positions relative to the content, the device can revise the displayed perspective of the content based on the definition file. When a user of a device interacts with the content, the device can determine how the content will change based on the definition file, and provide information describing the interaction to other devices displaying the content so that all devices and adjust the display.

 

In some embodiments, several devices can provide content to be displayed simultaneously by some or all of the devices. For example, several devices on a same team can display shared content. As another example, several devices playing in opposition can each provide opposing content to be displayed (e.g., two players provide opposing avatars that fight each other).

 

Patent Credits and Some Closing Thoughts

 

Apple's patent application that was first publically published today was originally filed in Q3 2010 by inventors Graeme Devine and Jeff Ruediger.

 

While we're on the topic of gaming, I'll take a moment to point out a key line that came out of Apple's special iPad event last week that's worth noting. During Epic Games demo, Mike Caps, President of Epic stated that "this new device actually has more memory and higher screen resolution than the Xbox 360 or PlayStation 3. So these guys are redefining mobile gaming, again."

 

The timing of that statement was interesting in light of the news that Microsoft's next generation Xbox is likely to be diskless. That's right, no disk, no Blu-ray. It may offer a MicroSD card short term and leap to the Cloud down the road. To get the bigger picture here, you should also read NeilsonWire's latest report titled "Trends in U.S. Video Gaming – the Rise of Cross-Platform." The gaming console is in 56% of homes and is likely to be the device that will eventually merge all of your entertainment content for the home via your TV.

 

Although Nielson didn't mention Apple in any way, I think that we must remember Mike Cap's statement that I just quoted. And with the iPad being officially classified as a TV this week in Canada's IP Office, you kind of get the sense that the concept of a gaming console is about to turn on its head. Apple's iPad has the volume that simply crushes both the Xbox and PlayStation 3 Combined. And if the gaming console of tomorrow is going ultrathin, look no further than the iPad. Apple has a number of patents covering a possible game controller coming to market and here's one that you could check out for yourself.

 

When Steve Jobs introduced the iPhone in 2007, he presented a slide of a breakdown of 2006 worldwide unit statistics to make his point why Apple had chosen to introduce the iPhone as opposed to a gaming console that many wanted at that point in time. Apple kept their eye on the prize and went on to win the new smartphone market.

 

Yet as in any revolution, it all boils down to timing. Apple could take Apple TV and crank up the gaming volume with an A6 processor this year or just launch a specialized iPad for gaming that has a slightly larger bezel for game controls. No matter how Apple decides to take gaming on, I think it's about time that Microsoft and Sony begin freaking out; because in a flash their consoles could be road kill.

 

Notice: Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. Apple's patent applications have provided the Mac community with a clear heads-up on some of Apple's greatest product trends including the iPod, iPhone, iPad, iOS cameras, LED displays, iCloud services for iTunes and more. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.