On August 18, 2011, the US Patent & Trademark Office published a patent application from Apple that reveals one of the next chapters for Apple's "Map + Compass" iOS feature. In the future, you'll be able to capture an intersection on your iPhone's camera and do a search based on that image. If you're looking for the Golden Gate Bridge while visiting San Francisco, Apple's Maps + Compass feature will be able to superimpose the directions from your current location to where you want to go. It will include turn by turn instructions according to Apple's documentation. Apple's hire of founder and CEO of PlaceBase, an advanced digital mapping firm, illustrates just how eager they are in advancing Maps + Compass for iOS devices with augmented reality. In fact, in typical Apple fashion, their new augmented application features will go far beyond just software. Apple has clearly indicated that they're developing a new interactive augmented reality 3D display that is considered to be a smart transparent display. This is what creates Apple's "Magic" product feature declarations when introducing a new product feature that drives the competition and critics crazy. Now you know that it's magic for very sophisticated reasons. It's a perfect blend of advanced hardware and software that few could match.
A Look at the Short Comings of Today's Augmented Reality Programs
Augmented reality systems supplement reality, in the form of a captured image or video stream, with additional information. In many cases, such systems take advantage of a portable electronic device's imaging and display capabilities and combine a video feed with data describing objects in the video. In some examples, the data describing the objects in the video could be the result of a search for nearby points of interest.
For example, a user visiting a foreign city could point a handheld communication device and capture a video stream of a particular view. A user could also enter a search term, such as museums. The system could then augment the captured video stream with search term result information related to nearby museums that are within the view of the video stream. This allows a user to supplement their view of reality with additional information available from search engines.
However, if a user desires to visit one of the museums, the user must switch applications, or at a minimum, switch out of an augmented reality view to learn directions to the museum. However, such systems could fail to orient a user's with a poor sense of direction and force the user to correlate the directions with objects in reality. Such a transition is not always as easy as it might seem. For example, an instruction that directs a user to go north on Main St. assumes that the user could discern which direction is north. Further, in some instances, street signs might be missing or indecipherable, making it difficult for the user to find the directed route.
Apple states that the challenges derived from the shortcomings as note above could be overcome using the present technology with their latest invention.
According to Apple, by interpreting the data describing the surrounding areas, the device could determine what objects are presently being viewed on the display. The device could further overlay information regarding the presently viewed objects, thus enhancing reality. In some embodiments, the device could also display search results overlaid onto the displayed video feed. Search results need not be actually viewable by a user in real life. Instead, search results could also include more-distant objects.
In one form of interaction, a device could receive an input from the user requesting directions from a present location to a selected search result. Directions could be overlaid onto the presently displayed video feed, thus showing a course and upcoming turns. As the user and associated device progress along a route, the overlaid directions could automatically update to show the updated path.
In some embodiments the display could also include indicator graphics to point the user in a proper direction. For example, if the user is facing south but a route requires the user to progress north, "no route" would be displayed in the display because the user would be looking to the south but the route would be behind him or her. In such instances, an indicator could point the user in the proper direction to find the route. In some embodiments, multiple display views could be presented based on the orientation of the device.
In practice, a user points their iPhone to capture and display a real-time video stream of a view. The iPhone detects a geographic position, camera direction, and tilt of the image capture device. The user sends a search request to a server for nearby points of interest. The iPhone receives search results based on the search request, geographic position, camera direction, and tilt of the iPhone. The iPhone visually augments the captured video stream with data related to each point of interest. The user then selects a point of interest to visit. The iPhone visually augments the captured video stream with a directional map to a selected point of interest in response to the user input.
Apple's patent covers a method of augmenting a video stream of an iPhone's present surrounding with navigational information. The user could instruct the iPhone to initiate a live video feed using an onboard camera and display the captured video images on the display. By polling a Global Positioning System (GPS) device, a digital compass, and optionally, an accelerometer, location, camera direction, and orientation information could be determined. By using the location, camera direction, and orientation information, the device could request data describing the surrounding areas and the objects therein. In some embodiments, this data includes map vector data. This could be requested from an onboard memory or a server. In theory, it could be an iCloud related server, as that is covered in other Apple related patents listed in this report. The data describing surrounding areas could further be requested in conjunction with a search request. The search request could also include a request for information about nearby places of interest.
Future "Maps + Compass" Feature to Utilize Augmented Reality
Apple's patent FIG. 1 is a screenshot illustrating an augmented reality embodiment. As illustrated, an iPhone has captured an image 102 of the northwest corner of the intersection of Dolores St and 17th St. using its camera to display the image. In this way, the iPhone's display could function as a viewfinder. As illustrated, the captured image 102 has been augmented with information corresponding to points of interest 104, 106 and street labels 110, 112.
In this illustrated embodiment, the user has entered a search term "parks" in search bar 108 to conduct a search for nearby parks, i.e., a specific type of point of interest. Using map data that describes the area surrounding the present location of the device and the points of interest located in the surrounding area, the device augments the displayed image with additional information. In this instance, the iPhone displays points of interest described by the data that are displayed in the viewfinder (such as Dolores St. 110 and 17.sup.th St. 112) or within a field of view and range from the geographic position of the device but that are obstructed by other in-screen objects, e.g., Golden Gate Park and Buena Vista Park. While other parks might also be nearby, they are not shown because they fall outside the field of view of the device. However, the user could locate these parks by panning the device around the intersection, in which case those parks would appear on the screen.
In the captured image the iPhone augments the captured image with bubbles showing the relative geographic position of "Golden Gate Park" and "Buena Vista Park" within the captured image. This allows the user to determine a general direction to a point of interest. A user could then select a point of interest, e.g., by selecting the "Buena Vista Park" point of interest information bubble, e.g., by touching the point of interest information bubble with a finger or stylus.
At the end of the day, Apple is describing the advancement of their "Maps + Compass" feature with Augmented Reality. It's a natural progression for this iOS device feature.
Feature: Points of Interest
Points of interest could be any map feature, but most often a point of interest could be a map feature that is identified as result of a search for a category of such map features. For example, a point of interest could be a park when a user searches for nearby parks. Likewise a point of interest can be places, buildings, structures, even friends that could be located on a map, when the point of interest is searched for. In some instances a point of interest is not necessarily identified as a result of a search. A point of interest could also be a map feature that is identified by the present system because it could be viewed in the captured image. In short, a point of interest can be any map feature for which the user has an interest.
Apple's patent FIG. 2 illustrates search results for point of interest results for nearby parks based on geographic position and also illustrates how a range and field of view correspond to the results displayed in the viewfinder. An iPhone captures a video stream of the view as shown in FIG. 1. The iPhone detects the geographic position, camera direction, and tilt of the handheld communication device.
The geographic position of the handheld communication device could be determined using GPS coordinates or using triangulation methods using cell phone towers. In yet another example, a blend of GPS coordinates and triangulation information could be used to determine the position of the device.
The Schematic Maps Feature
Apple's Jaron Waldman states that in some embodiments, the route could be represented as a schematic map, i.e., a simplified map that includes only relevant information for the user in an easy-to-read format. "A schematic map could be thought of as similar to a subway map one would see on a subway train." This feature was described in last week's patent report titled "Multiple Carrier Profiles & Schematic Maps may be on iPhone's Roadmap."
Visually Augmenting a Map with a Suggested Route
Apple's patent FIG. 3 shown below illustrates a captured image that has been visually augmented with route data to a selected point of interest. In this example, a user has selected the "Buena Vista Park" point of interest and in response the iPhone has visually augmented the captured image 302 with a directional map 310 to the selected point of interest, i.e., "Buena Vista Park".
The route shows a direction 312 that the user must travel on Dolores St. to begin travelling to reach "Buena Vista Park." The directional map further indicates a turn 314 that the user must take, i.e., a turn left onto Duboce Ave. from Dolores St. In the illustrated example, the map is shown overlaid onto Dolores St.
The route guides the user with complete navigation illustrations to reach "Buena Vista Park," including any required turns. In some embodiments, the route could be represented as a schematic map, i.e., a simplified map that includes only relevant information for the user in an easy-to-read format as discussed above. Apple's patent FIG. 4 is a flow chart illustrating an exemplary method of preparing and displaying an augmented reality map.
Apple's patent application was originally filed in Q4 2009 by inventor Jaron Waldman. Jaron Waldman was the CEO and founder of PlaceBase before joining Apple's Geo Team.
Other Noteworthy Patent Applications Published Today
Apple has applied for a continuation patent under 20110199171 for magnet detents, originally filed in 2007. Apple was granted a patent for this in May 2011. Apple has filed for another continuation patent that dates back to 2007 concerning "Numbers" spreadsheet program under 20110202823 titled "Pasting a Set of Cells." And lastly, we see that another Apple patent application has surfaced (20110201213) regarding a low profile plug for handheld devices. This adds to their previous patent on this matter.
Notice: Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. Apple's patent applications have provided the Mac community with a clear heads-up on some of Apple's greatest product trends including the iPod, iPhone, iPad, iOS cameras, LED displays, iCloud services for iTunes and more.
About Comments: Patently Apple reserves the right to post, dismiss or edit comments.
Extra Feature: 3D Printer Video
If you've never seen this video before, then check it out. It's truly one of the wildest inventions that I've seen in some time. And no, it's not from Apple.
About this Technology: 3D printing is a form of additive manufacturing technology where a three dimensional object is created by laying down successive layers of material. 3D printers are generally faster, more affordable and easier to use than other additive manufacturing technologies. 3D printers offer product developers the ability to print parts and assemblies made of several materials with different mechanical and physical properties in a single build process. Advanced 3D printing technologies yield models that can serve as product prototypes.
Here are a Few Great Community Sites covering our Original Report
MacSurfer, Blogrunner, Twitter, Facebook, Newapples Russia, Apple Investor News, Google Reader, UpgradeOSX, Rotter.net Israel, The Eponymous Pickle, TechWatching, Macnews, iPhone World Canada, CBS MarketWatch, MacDailyNews, 9to5 Mac, TiP Today's iPhone, AppAdvice, CNET, ReadWriteWeb, Mac Life Germany, MacMagazine Brazil, Techmeme, SlashGear, ITespresso Spain, iFun iPhone Germany, Mobile Entertainment, Italiamac Italy, TUAW, PrePosterous, International Business Times Spanish, Telefonino Italy, and more.