Today the US Patent & Trademark Office published a patent application from Apple that relates to the future use of Virtual Paper on devices that will be able to crumple, flip, fold, bend while keeping the graphics intact in varying positions as the virtual paper changes angles and perspectives. Virtual paper could include mixed reality content in 2D, 3D and animation where the image could be constantly shifting depending on a particular state that it's in. This is pretty cool stuff that's now in-the-works.

The patent is not really meant for consumers, though we'll eventually be able to enjoy virtual paper on our future head-mounted devices, iPhones and iPads.

The filing appears to be more for developers or perhaps it's Apple's work that will be coming to future developer tools, as they reference things like "each of the 3D toolbox objects."   

In the big picture, Apple's patent covers devices, apparatuses, systems, and methods for intermingling stereoscopic and conforming virtual content to a virtual paper.

Apple notes that in contrast to previously available computer-generated reality (CGR) systems, various embodiments disclosed in their patent filing provide a multi-dimensional CGR environment. In some embodiments, the multi-dimensional CGR environment includes a display of a virtual paper that is represented as a bounded surface or a bounded region.

For example, while the 3D virtual content is displayed such that the 3D virtual content is bounded within the perimeter of the virtual paper, the 3D virtual content is able to protrude out from a first side of the virtual paper, and the display of the 3D virtual content is based on the second set of world coordinates.

The 2D virtual content, on the other hand, is displayed as conforming to the surface of the virtual paper based on the first set of world coordinates. In some embodiments, a second side of the virtual paper, which is the opposite of the first side of the virtual paper (e.g., the backside), is displayed with a rasterized conforming representation of the 3D virtual content, e.g., a blur effect or shadow of the 3D virtual content, along with a rasterized conforming representation of the 2D virtual content.

In some embodiments, a native user CGR environment is a CGR environment in which a user or an avatar representing the user is located. For example, the CGR environment (#400) includes a user (#401) or an avatar representing the user.  As such, the scene depicted in FIG. 4A below can be a view of the CGR environment from the perspective of the user or a bystander.

Additionally, we see in Patent FIG. 4A below the 2D text "Matrix" (#410) appears to be floating on the surface of the front side of the virtual paper (#405), and the 3D chicken (#420) appears to be inside the virtual paper.

As such, the 2D text "Matrix" appears to be in front of (e.g., overlaid on) the 3D chicken from the perspective of the user. Accordingly, the 2D text "Matrix" occludes part of the crest of the 3D chicken inside the virtual paper.

Apple's patent FIG. 4B illustrates the backside of the virtual paper with the chicken, the bounded surface (#405) appears to be translucent or semitransparent, such that we can see the outline of a blurred chicken image with just the tip of the beak present on the far right of the image to show you that graphics are following the movement of the virtual paper.

The patent then goes 10 miles deep into the lighting, rasterized images, spheres, pixel depths that could lose the average reader.

As show in FIGS. 4C-4E during the transformation of the display of the virtual paper, the display of the virtual contents associated with the virtual paper also transforms in response to transforming the contour of the virtual paper. For example, parts of the 3D-chicken protrude from one side of the virtual paper as the virtual paper or virtual page is turning.  

Lastly, in Apple's patent FIGS. 4L and 4M, they note that in addition to the stereoscopic virtual object of the 3D chicken being displayed in a web page, the 3D chicken's head could be animated by turning, blinking and so forth.

Apple's patent illustrates Virtual Paper being used in future applications such as Apple Maps, Apple Music and other apps relating to travel and obviously books, magazines and so forth.  In patent FIG. 5i below, Apple tells us that a consumer viewing an image on an mall ad or magazine will be able to turn the boat around by gesture in order to view different parts of the ship and so forth. 

More specifically, Apple notes that in some embodiments, properties of the ocean liner (#530), e.g., distance, direction, orientation, scale, moving speed, etc., can be further adjusted in response to an input. For example, in response to an input directed to the bounded region (#505), the ocean liner can be turned around within the bounded region. As such, the stern (or the rear) of the ocean liner would be closer to the user, and the user can examine the stern of the ocean liner in close distance.

Apple's patent application 20200302698 that was published today by the U.S. Patent Office was filed back Q1 2020 with some of the work incorporated into this patent dating back to Q1 2019. For developers wanting to review the patent filing deeper could do so here.

Considering that this is a patent application, the timing of such a product or developer tool to market is unknown at this time.