Now that we know that Apple Vision Pro is coming to market next year, we're one step closer to realizing Apple's invention regarding Virtual Paper. It first came to light in 2020 and on Tuesday, the U.S. Patent and Trademark Office officially granted Apple their second patent for this invention.

Virtual paper will one day be able to include mixed reality content in 2D, 3D and animation where the image could be constantly shifting depending on a particular state that it's in. Virtual Paper will be able to be crumpled, flipped, folded and bended while keeping the graphics intact in varying positions as the virtual paper changes angles and perspectives as presented in our cover graphic. This will likely lead to next-generation books and magazines where content could be animated on a page to give a story or an ad a completely new experience.

Apple notes in their granted patent that while 3D virtual content is displayed such that the 3D virtual content is bounded within the perimeter of the virtual paper, the 3D virtual content is able to protrude out from a first side of the virtual paper, and the display of the 3D virtual content is based on the second set of world coordinates.

The 2D virtual content, on the other hand, is displayed as conforming to the surface of the virtual paper based on the first set of world coordinates. In some embodiments, a second side of the virtual paper, which is the opposite of the first side of the virtual paper (e.g., the backside), is displayed with a rasterized conforming representation of the 3D virtual content, e.g., a blur effect or shadow of the 3D virtual content, along with a rasterized conforming representation of the 2D virtual content.

Virtual paper not only will improve how the virtual content is displayed in the CGR environment within Apple Vision Pro, but also how the virtual content is interacted with in the CGR environment.

Apple's patent FIG. 4A below illustrates an example of a CGR environment #400 in which a first side (e.g., a front side) of a virtual paper #405 is displayed in accordance with some embodiments. In some embodiments, the CGR environment is a native user CGR environment. In some embodiments, a native user CGR environment is a CGR environment in which a user or an avatar representing the user is located. For example, the CGR environment includes a user #401 or an avatar representing the user. As such, the scene depicted in FIG. 4A can be a view of the CGR environment from the perspective of the user  or a bystander.

As shown in FIG. 4B below, the backside of the bounded surface #405 includes the conforming representation #420-B of the 3D chicken #420 from the front side of the bounded surface of the Virtual Paper. As a result, the bounded surface appears to be translucent or semitransparent, such that a blurred chicken image is displayed on the backside of the bounded surface within the perimeter of the backside of the bounded surface.

In addition, the text on the virtual paper #405 also transforms, e.g., the text “Chicken” is slanted in FIG. 4C; In 4D, the text isn't seen but the chicken's profile shifts from FIG. 4C; and in FIG. 4E we see that as the paper is twisted, we see the shadow of the chicken on the backside of the virtual paper.

In FIGS. 4L and 4M, without user inputs directed to the virtual paper, there is no transformation of the virtual paper and/or the conforming virtual objects (text #462 and #464). While the virtual paper remains stationary, the 3D chicken #460 animates, e.g., the head position of the 3D chicken in FIG. 4L differs from the head position of the 3D chicken 460 in FIG. 4M. In other worlds, the display of the 3D chicken is transformed based on its own set of world coordinates (e.g., the second set of world coordinates (X.sub.2, Y.sub.2, Z.sub.2.

In patent FIG. 5i above, Apple tells us that a consumer viewing an image on an mall ad or magazine will be able to turn the boat around by gesture in order to view different parts of the ship and so forth.

Apple's patent illustrates Virtual Paper being used in future applications such as Apple Maps, Apple Music and other apps relating to travel along with books, magazines, ads and more.

In their second granted patent, Apple has added 22 new patent claims. For more details, review U.S. granted patent 11748966. This patent was also published by the European Patent Office on August 30 under #EP4235381.