The U.S. Patent and Trademark Office officially published a series of 49 newly granted patents for Apple Inc. today. In this particular report we cover Apple's latest patent win relating to Augmented Reality. It's an invention that may have been incorporated into Apple's ARKit being that Apple acquired Metaio's intellectual property and engineering for their leadership in augmented reality. Today's patent is about a method of creating AR for use with devices like Apple's iPhone, iPad and future smartglasses which this patent describes.

Granted Patent: Method for Representing Virtual Information in a Real Environment

Apple's newly granted patent covers their invention relating to a method for representing virtual information in a view of a real environment comprising the steps of providing a system setup with at least one display device, wherein the system setup is adapted for blending in virtual information on the display device. The invention also relates to a computer program product comprising software code sections for performing the method.

Augmented reality (AR) systems are known to enhance information of a real environment by providing a visualization of overlaying computer-generated virtual information with a view of the real environment or a part of the real environment. The virtual information can be any type of visually perceivable data such as objects, texts, drawings, videos, or their combination. The view of the real environment or the part of the real environment, as understood herein, could be perceived as visual impressions by user's eyes and/or be acquired as one or more images by a camera, e.g., worn by a user or attached on a device held by a user.

The overlaid or blended in virtual information may be, in principle, various items of virtual information. For example, an item of virtual information which could enhance information of a real environment may be a point of interest, as for example known in map or navigation applications. A point of interest (POI) may represent a location of a real object of the real environment (e.g., a building or a landmark) and often includes digital content that is related to the real object. For instance, the location is a global location (e.g., a geo-coordinate such as a 2D coordinate of longitude and latitude, or a 3D coordinate of longitude, latitude and altitude) or a post address (e.g., a floor number, street, postcode, country). The post address and the global location could be converted to each other. The digital content of the POI could contain various data, such as a name, description, and contact related to the real object.

One major function of augmented reality systems is to overlay items of virtual information, such as points of interest (POIs), to a view of the real environment. This is particularly useful and popular in location-based (mobile) augmented reality applications, such as tour guidance for exploring the urban environment or even enhancing indoor mapping.

In addition, users could use an augmented reality application to overlay POIs to a view of the real environment when they are in new places and want to obtain information about things they see. In augmented realty systems, the POI information has to be represented in the real environment or the part of the real environment such that it satisfies desired visual perception and usability.

Most data sources for POI information provide the information in relation to a point in the real world, not as a 3d model with more than one point or vertex. This invention is well suited for handling the visualization of items of virtual information, which are described by latitude, longitude and optionally altitude and additional meta-information consisting of text and 2D image information.

Several methods have been developed for improving visual perception and usability of blending in or overlaying POI information to a view of a real environment in augmented reality applications.

A real object of the real environment often occupies a three-dimensional space which is a volume in 3D space, while a POI related to the real object represents a location which is a point in 3D space. The POI location could be assigned anywhere within the real object that the POI is related to. Therefore, the real object could be visible from the view point, while the POI has a longer view distance than the estimated depth along the view direction to the real object. For instance, a building (e.g. a shopping mall) is visible in a view of the real environment, while the view distance of a POI related to the building is longer than the depth from the view point to the exterior surface (e.g. wall) of the building.

Illustrated below, Apple's Patent FIG. 8 shows us an exemplary image of the real environment acquired at the viewing point with the field of view; FIG. 9 shows the scene of FIG. 8 in which Points of Interest (POIs) are opaquely overlaid to the image.

The granted patent further notes that the digital content or items of virtual information can be any type of visually perceivable data such as objects, texts, drawings, videos, or their combination. The view of the real environment or a part of the real environment is captured by a capture device. For example, the view of the real environment could be captured as visual impressions by user's eyes or acquired as one or more images by a camera worn by a user or attached on a device held by a user (Think iDevice).

Another alternative display device could be a head-up display and yet another alternative points to smartglasses.

The patent notes that "The overlaid or blended in information of the items of virtual information, e.g. POIs, and the real environment can be seen by the users in a well-known optical see-through display having semi-transparent glasses. The user then sees through the semi-transparent glasses objects of the real environment augmented with the virtual information of the POIs blended in the glasses

Today's granted patent that Apple inherited from their Metaio acquisition supports Apple's earlier patent from Metaio relating to AR smartglasses as noted below which is in line with today's granted patent.

Apple's granted patent 9,922,446 was originally filed in Q2 2014 and published today by the US Patent and Trademark Office. To view other Apple AR patents, see our Augmented and Virtual Reality Archive.

Patently Apple presents only a brief summary of granted patents with associated graphics for journalistic news purposes as each Granted Patent is revealed by the U.S. Patent & Trademark Office. Readers are cautioned that the full text of any Granted Patent should be read in its entirety for full details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or behavior will result in being blacklisted on Disqus.