In June 2015 Apple acquired Germany's Metaio, a pioneer in Augmented Reality (AR) and thermal touch technology. In 2018 Apple was granted their first smartglasses patent in context with AR.  In the last month the US Patent & Trademark Office published a continuation patent application from Apple that came from Metaio's patent portfolio that they acquired.

Apple's patent lists Michael Kuhn and Peter Georg Meier as the inventors. Meier is listed on LinkedIn as the prior owner of Metaio who is now Apple's Director – Algorithm Lead in the Technology Development Group. Kuhn is now a Software Developer Manager at Apple.  

As with their previous smartglasses patent, this former Metaio patent now owned by Apple, covers "data glasses (e.g. a so-called optical see-through data glasses or video see-through data glasses)" and beyond.

For instance, Apple's patent FIG. 5 below shows an exemplary scene of a real environment having virtual objects placed therein, without pose improvement having been effected. The patent figure covers an iDevice with a built-in camera.

Apple notes in their filing that the invention can also be installed, or carried along, in vehicles, aircraft or ships, making use of a monitor, HMD or a head-up display or a smartphone.

Because the technology uses GPS, it could also be applied to a "Street View" Maps application, the filing notes. Our cover graphic from a Metaio patent shows the "Street View" app applying to AR smartglasses.

Being that it's technically a "continuation patent," Apple did in fact change some of the patent claims to better reflect what Apple wants protected in this invention.

As noted above, GPS plays a role in this AR invention in order for smartglasses, iDevices and Heads-up Displays to receive and present users with realtime data. Apple listed a number of new patent claims to protect this aspect of the invention as noted below:  

Patent Claim #2:  The method of claim 1, further comprising: capturing the image at a camera of a mobile device; and receiving global positioning system (GPS) data from a GPS sensor of the mobile device, wherein the image pose is determined based on the GPS data.

Patent Claim #3: The method of claim 1, further comprising: capturing the image at a camera of a mobile device; and receiving an identifier from a wireless local area network adapter of the mobile device, wherein the image pose is determined based on the identifier.

Patent Claim #4: The method of claim 1 … the reference coordinate system is a geographic global coordinate system.

An Apparatus

Most importantly and key to the new patent claims is Apple's introduction of "an apparatus" that wasn't a part of Metaio's original granted patent claims. This will apply to any apparatus that Apple has listed in the patent such as iDevices, smartglasses, an HMD and more.

Patent Claim #13: An apparatus comprising: a location sensor; a camera; one or more processors; and a memory storing computer readable code executable by one or more processors to: initiate capture of an image by the camera; determine an image pose of the image in a reference coordinate system based on location data from the location sensor; receive, from a computing device, a reference image and a reference pose of the reference image in the reference coordinate system; match features of a real object depicted in the image to corresponding features of a real object depicted in the reference image; generate an updated image pose based at least in part on the reference pose in response to determining that the features of the real object depicted in the image match the corresponding features of the real object depicted in the reference image; determine an overlay position of a virtual object in the image based on user input; and initiate transmission, to the computing device, of a request to store an object pose of the virtual object in the reference coordinate system based on the updated image pose and the overlay position.

Patent Claim #14: The apparatus of claim 13, wherein the location sensor includes a global positioning system (GPS) sensor and the location data corresponds to GPS data.

Apple's "continuation patent" application was filed in September 2018 with the original patent filed at an earlier period (under Metaio). In fact Metaio was granted a patent for this invention back in 2015. The timing of such a product to market is unknown at this time.