Apple acquired Metaio the creator of 'Thermal Touch' and a new Augmented Reality Interface for Wearables and beyond back in 2015. Their technology is thought to be behind Apple's push into augmented reality and ARKit. This year a Metaio patent application surfaced under Apple for moving furniture in augmented reality. Apple was also granted a patent for indoor navigation that covered new capabilities for a future iDevice camera allowing it to recognize building names or paintings and then adding AR identifying markers on the user's iDevice photos. Today another original Metaio patent application under Apple has surfaced relating augmented reality. More specifically it covers a method for representing points of interest in a view of a real environment on a screen of an iPhone with interaction functionality. The buzz is that the patent covers AR smartglasses as noted in our cover graphic, something that Apple has been adding to a series of new and updated trademarks of late ( one, two, three and four).

Augmented Reality for iDevices including Smartglasses

It is known that Augmented Reality (AR) systems could enhance information of a real environment by providing a visualization of overlaying computer-generated virtual information with a view of the real environment or a part of the real environment. The virtual information can be any type of visually perceivable data such as objects, texts, drawings, videos, or their combination. The view of the real environment or a part of the real environment could be perceived as visual impressions by user's eyes and/or be acquired as one or more images captured by a camera held by a user or attached on a device held by a user.

Apple's patent FIG. 7 presented above illustrates a depiction (left) according to which a mobile device, such as a mobile phone, is held at its vertical position, and a depiction (right) of an exemplary view displayed on the touchscreen of the mobile device; FIG. 8 shows a flowchart of an embodiment of the invention presenting one or more POIs on a touchscreen of a mobile device, such as a mobile phone, equipped with a camera.

Apple's patent further notes that a point of interest (commonly referred to as "POI") is known in the art to represent a location or a real object (e.g., a building, a landmark, a moving car) in a real environment. A POI often has associated digital content that is related to the location or the real object. The digital content could be, for instance, audio information, video information, pictures, textual information, 3D representations or their combinations.

Representing points of interest (POIs) in a view of a real environment is particularly useful and popular in location-based (mobile) augmented reality applications, such as tour guidance for exploring the urban environment. For example, users may use augmented reality to see visual information related to POIs overlaid with the view of the real environment when they are in unknown places or want to obtain information about things they see.

According to a first aspect, there is provided a method for representing points of interest in a view of a real environment displayed on a screen of a mobile device with a functionality for interaction with a user, which comprises the steps of: capturing an image of the real environment or a part of the real environment using a camera, determining at least one point of interest related to the real environment, determining an image position of the at least one point of interest in the image, displaying at least part of the image on at least part of the screen, overlaying a computer-generated indicator with the at least part of the image on the screen at a screen position according to the image position of the at least one point of interest.

AR Centric Smartglasses / Head-Mounted Display

Apple notes here that this embodiment is particularly useful when using a head-mounted display comprising the camera and the screen. For example, the head-mounted display is a video-see-through head-mounted display (HMD). It is typically not possible for the user to touch the head-mounted screen in a manner like a touchscreen. However, the camera that captures an image of the real environment may also be used to detect image positions of the user's finger in the image. The image positions of the user's finger could be equivalent to touching points touched by the user's finger on the touchscreen.

Apple's patent FIG. 10a below shows us an example of state of the art for representing POIs in the view of a real environment on a semi-transparent screen of a head-mounted display.

According to a second aspect, there is provided a method for representing points of interest in a view of a real environment on a semi-transparent screen of a mobile device with a functionality for interaction with a user, comprising the steps of: determining at least one point of interest related to the real environment, determining a position of the at least one point of interest relative to the view, blending in a computer-generated indicator in at least part of the view on the semi-transparent screen at a screen position according to the position of the at least one point of interest relative to the view, blending in a computer-generated virtual object related to the at least one point of interest on the semi-transparent screen at a screen position determined according to the screen position of the computer-generated indicator and which is adjacent to a bottom edge of the semi-transparent screen, and blending in a visually perceivable relation indication indicative of a relation between the computer-generated virtual object and the computer-generated indicator. Preferably, the mobile device may perform an action related to the at least one point of interest if at least part of the computer-generated virtual object blended in on the semi-transparent screen is overlapped by a user's finger or device held by the user.

Semi-Transparent Multidimensional User Interface for a Future Desktop and Application Environment 

With so much emphasis on semi-transparent screens (as noted above), it should be noted that today an Apple patent application has surfaced specifically talking about semi-transparent displays. The application titled "Rendering Semi-Transparent UI Elements," covers a multidimensional desktop and application environment as noted in FIG. 8 below. Having a semi-transparent user interface would seem to magically work with a future smartglasses device. You could check out the details of the second patent here.

Apple's patent application 20170213393 was filed back in April 2017 with a history dating back to 2013. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.