Today the US Patent & Trademark Office published a patent application from Apple that relates to an augmented reality device to provide next generation navigation for Maps. The AR device could be built into future vehicles (a Project Titan vehicle or not). Alternatively, the AR device could be a pair of smartglasses or a next-gen iPhone sitting on the vehicle's dashboard. The devices must have  an advanced AR camera built into them in order to enable the use of this proposed augmented reality mapping system.  

Apple patent application states that an augmented reality device can enhance and improve the ease at which a device can convey lane guidance information to a user that is traveling along a route.

The device can capture a series of images (e.g., video footage) in front of the augmented reality device that closely represents what is actually perceived by the user, and display the captured series of images in real time, e.g., on a screen of the device or as a projection.

Concurrently, the device can superimpose a navigational layer over the displayed captured images to guide the user into one or more lanes that would best position the user to proceed along the route.

To enable this functionality, the augmented reality device can be configured to communicate with one or more global positioning system (GPS) satellites and one or more route servers. The GPS satellites can utilize triangulation to provide the location of the device, and the route server can utilize one or more databases to provide route information regarding the route of travel between an origin location and a destination location (or to each intervening location between the origin and destination locations).

The route information can include, but is not limited to, road information (e.g., identification of roads along the route, what segments of those road will be traveled, the geographical position of those roads, the geographical position of lanes within each road, and in what order will the roads be traveled), turn information (e.g., what turns are needed along the route and in what order), lane information (e.g., number and position of lanes for each road segment), and distance information (e.g., distance to travel on each segment of road).

In some embodiments, the augmented reality device can also be configured to communicate with any other positioning systems, such as, but not limited to, wireless fidelity (WiFi)-based positioning systems, cellular based positioning systems, satellite-based positioning systems, and any other global navigation satellite systems (GNSS).

With this information, the augmented reality device can analyze the video footage to determine the number of available lanes. The device can then correlate the video footage with the location of the device, which can be taken as the location of the vehicle, and identify one or more designated lanes from the one or more available lanes in which the vehicle should be traveling to stay in the route. The device can display the series of images along with a navigational layer over the series of images to guide the user into the correct lane, or a plurality of correct lanes, if applicable.

In some embodiments, the navigational layer can block out regions (e.g., lanes, curbs, sidewalks) of the captured images where the vehicle is not intended to travel, and leave unblocked regions (e.g., lanes) where the vehicle is intended to travel. That way, the user can clearly understand which lane he or she should be in. In additional embodiments, the navigational layer can include an indicator positioned over the unblocked region.

The indicator can be a single guidance arrow or a series of guidance arrows that point toward the direction in which the user should travel. For instance, the indicator can be a straight arrow that points diagonally to the right, indicating that the user should switch lanes to the right.

In another instance, the indictor can be a series of multiple diagonal arrows that blink in a sequential order indicating the same. The indicator can blink to convey a degree of urgency at which the user should perform the indicated task.

Additionally, in some embodiments, a pop up window can appear in the navigational layer to provide more information to the user about the upcoming turn. For example, the pop up window can be a birds eye view of a turn that the user is about to take. That way, the user can better understand what is about to happen.

Superimposing a navigational layer over video footage of what the user can see helps clearly communicate navigational and lane guidance information to the user. The augmented reality device can provide significant improvements in navigation and lane guidance over conventional devices that merely provide symbols to guide a user along a route.

Devices that provide navigation using symbols require the user to decipher the symbols in real time and immediately apply them to what the user is perceiving, which may not always be easy to do.

In Apple's patent FIG. 3 below we're able to see a snapshot illustrating an exemplary implementation of an augmented reality device in a vehicle; FIG. 2 is a snapshot illustrating an exemplary navigational interface of an augmented reality device.

More specifically, Apple's patent FIG. 3 above illustrates an exemplary implementation (#300) of an augmented reality device (#302 highlighted in yellow).

The augmented reality device can be used in a vehicle that is capable of being driven or otherwise controlled by a user. In some embodiments, the augmented reality device can include a camera (not shown) for capturing images of front area (#306), or can include hardware and software for communicating with a separate camera positioned to capture images of the front area.  

If the AR device includes a camera, then the device can be positioned against a windshield or anywhere within the vehicle so that the device can perceive a front area of the vehicle. In some embodiments, the device can be integrated into vehicle 304.

To enable the augmented reality device to provide navigational and/or lane guidance functions, the device can be configured to communicate with external devices, such as one or more GPS satellites (#308) and a route server (#310) through a wireless network (#312).

The augmented reality device can be configured to receive location data from a GPS satellite and route information from route server, analyze captured images to determine available lanes in front of the vehicle, apply the route information and location data to the analyzed images, and then superimpose a navigational layer over the captured images while the captured images are concurrently displayed to the user, thereby providing an augmented reality interface to a user (shown in FIG. 2).

While the AR device could be a future custom device, Apple notes that "the device could be an iPhone [with a next gen AR backside camera that has already been forecasted] or a pair of smartglasses."

In a somewhat humorous note describing the AR interface of FIG. 2, Apple notes that "By providing horizon (#212), the navigational layer can clearly indicate that the blocked region (#208) applies to the ground upon which the user is traveling and not the sky." Gee, thanks, I wouldn't want to make that mistake.

In addition to the blocked region and unblocked region (#210), the navigational layer can also include additional features such as one or more indicators and texts. For instance, the navigational layer can include an indicator 216 that visually communicates to the user the intended direction of travel. The navigational layer can also include text 218, such as the name of the street, road, or highway along which the user is currently traveling. Furthermore, the navigational layer can include additional road information, e.g., speed limit information 220, to inform the user what the maximum speed of travel is for the current road. The navigational layer can also include more route information, such as upcoming turn information in the form of a pop-up window.

Apple's patent FIG. 4 below is a block diagram illustrating an exemplary augmented reality system that can be implemented in an augmented reality device to enable the features of navigation and lane guidance using augmented reality.

Apple's patent FIG. 5A below is a simplified top-down diagram of a vehicle traveling along a segment of an exemplary route; FIGS. 5B and 5C are snapshots illustrating a navigational layer superimposed over captured images for an exemplary case where only one lane in a multi-lane segment of road is a designated lane.

Apple's patent FIG. 8A below is a simplified top-down diagram of a vehicle traveling along a segment of an exemplary route having an unusual turn; FIG. 8B is a snapshot illustrating an exemplary navigation layer including a pop-up window after a threshold distance has been crossed; FIG. 8C is a snapshot illustrating an exemplary navigation layer including a pop-up window as the vehicle in FIG. 8B encounters the U-turn, according to some embodiments of the present disclosure.

Apple's patent FIG. 9 above is a block diagram of a method for performing navigation and lane guidance.

Apple's patent application that was published today by the U.S. Patent Office was filed back in Q1 2019. Some of the work dated back to Q1 2018. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.