Today the US Patent & Trademark Office published a patent application from Apple that relates to future iDevices being able to view Augmented Reality Maps on a tabletop or desk.

We've seen demos of game environments transferred to a table top and various Apple product demos. Apple is promoting "SwiftShot" to create AR games that could play out on a tabletop today as explained by Apple here.  Now Apple wants to bring AR to Apple Maps with the same kind of viewable execution.

Apple Maps AR

Apple's invention provides for an augmented reality and/or virtual reality interface (e.g., in that movement of the device is used) that allows for interaction with a displayed map. The interface may be implemented in association with a map or navigation application. For example, the interface may be used by a tourist to easily and intuitively interact with a three dimensional map of an unfamiliar city, without requiring fingertip manipulation on the display. The interaction may be made by physically moving (e.g., translation or rotation) the mobile device upon which the map is displayed. For example, the view of the map may be zoomed in when the mobile device is moved toward a physical object in the same room as the user, and zoomed out when the mobile device is moved away the physical object. Such zooming in and out can be determined based on analysis of changes of size and relative placement of physical objects in images of a camera of the mobile device, where the virtual map objects have a specified relationship to the physical objects. Similarly, the view of map may be shifted left when the mobile device is turned left, right when the mobile device is turned right, and the like.

The physical movement of the mobile device may be determined using one or more sensors integrated into the mobile device (e.g., gyroscopes, accelerometers, compasses, etc.). In addition or alternatively, the physical movement of the mobile device may be determined by consecutive images taken by the physical camera of the mobile device. For example, a first image may be captured showing a room with a desk, a computer, and a bookshelf, where certain map objects (e.g., buildings, street names, etc.) can be displayed on a screen of the mobile device as if they were at about the location of the computer. The map objects can be displayed alone, or parts or all of a camera image of the physical world can also be displayed.

A second, subsequent image may be captured showing only the computer (e.g., encompassing more pixels of the image), indicating that the user has walked toward the computer, and thus moved toward the map objects. The physical movement of the physical camera of the mobile device toward the computer may be translated into a virtual movement of the virtual camera in the map application. In this case, the physical movement of the mobile device to be closer to the computer may be translated into a zoomed-in view of the three-dimensional map. Thus, movement of the mobile device can allow a user to control what parts of a map (e.g., a three-dimensional map) are displayed on the mobile device.

Embodiments of the present invention provide a number of advantages. New users of map applications, particularly those who may not be mobile device-savvy, may find traditional methods of manipulating maps to be difficult. For example, a user may be familiar with one-finger manipulations, such as entering an address, dropping a pin, moving the map, etc., but may be less familiar or unfamiliar with more complicated two-finger manipulations. Such two-finger manipulations may be required to rotate the map, zoom into the map, zoom out of the map, and the like.

So, some embodiments of the invention provide an interface for interacting with a displayed map that is easy and intuitive, allowing users to interact with the displayed map by moving the mobile device. For example, a user may rotate a displayed map by rotating the mobile device.

Apple's patent FIG. 1 below shows a block diagram depicting a mobile device implementing augmented reality; FIG. 2A presents a diagram depicting a user using a mobile device to register and view an augmented reality virtual map as if a three-dimensional map existed in a room.  

Apple's patent FIG. 2B above presents a diagram depicting a user using a movement-controlled user interface of the augmented reality virtual map to view the three-dimensional map from a different perspective with respect to FIG. 2A according to some embodiments of the invention.

Apple further notes in respect to FIG. 2A a diagram depicting a user (#202) using an iPhone to view a virtual map. Specifically, the user may use a physical camera of the mobile device at initial position (#205) to capture an image of the desk (#230). The iPhone may identify the desk as a suitable surface (e.g., a horizontal surface or flat surface) on which to overlay a map image including three dimensional map objects (e.g., building #215, tree #220, and bank #225). So, an iPhone may display the three dimensional map objects onto the desk as seen on the display of the mobile device.

Further, the building tree and bank may appear to be positioned on the desk such that the base of the objects appear to be positioned on the desk with the objects protruding from the desk. The dashed lines in FIG. 2A indicate that these map objects are only seen by a user when viewing a display of the mobile device.

Apple's patent application was filed in May 2016. Considering that this is a patent application, the timing of such a product to market is unknown at this time. A few of Apple's inventors listed on this patent include Storrs Hoen (Master Scientist/Engineer) and Aidan Zimmerman (Senior Product Design Engineer).