Apple updates 2 patents relating to a Future Interactive Vehicle Heads-Up Display & Virtual Paper used in Mixed Reality Environments
Today the US Patent & Trademark Office published two patent updates from Apple. The first covers a future interactive vehicle heads-up display that a user could use in-air gestures to control various interfaces, including games, in autonomous or semi-autonomous vehicles. The second patent relates to the use of Virtual Paper in AR/VR environments, giving virtual paper many dimensions beyond real world paper.
Apple Updates Two Interesting Patents
Future Vehicle Heads-Up Displays Interacting with Driver Hand Gestures
Aspects of Apple’s invention involve projecting an interactive scene onto a surface from a projecting object. In one particular embodiment, the interactive scene is projected from a vehicle and may be utilized by the vehicle to provide a scene or image that a user may interact with through various gestures detected by the system.
In addition, the interactive scene may be customized to one or more preferences determined by the system, such as user preferences, system preferences, or preferences obtained through feedback from similar systems. Based on one or more user inputs (such as user gestures received at the system), the projected scene may be altered or new scenes may be projected.
In addition, control over some aspects of the vehicle (such as unlocking of doors, starting of the motor, etc.) may be controlled through the interactive scene and the detected gestures of the users.
Apple’s patent FIG. 1 below is a diagram of a system for a vehicle to project an interactive scene onto a surface for interaction with a user of the projection system. In one particular embodiment, the projection system #100 is integrated into or otherwise associated with a vehicle, and more particularly an autonomous vehicle to allow a user of the projection system to interact with the vehicle to control one or more aspects of the vehicle.
Apple's patent Figure 3 above is a diagram illustrating projecting a welcoming interactive scene from a vehicle; Figure 4 is a diagram illustrating projecting an interactive numeric or phone keypad. In another example, a LIDAR system or other type of depth measurement device such as a RADAR system may be utilized to detect the user's position within the scene; Figure 5 is a diagram illustrating projecting a game interface interactive scene from a vehicle.
You could review our original 2018 patent report here that was first discovered in Europe for more graphics and details. Apple patent was granted in September 2021 under number 11,117,535. Apple’s updated patent published today interestingly scraps the original 23 patent claims for a single patent claim noted below:
New Patent Claim #1: ”A system for projecting an image onto a surface near a vehicle, the system comprising: a presence detector that detects the presence of a user within a distance threshold of a projection system integrated with a vehicle, the presence detector comprising a proximity sensor and a recognition system to determine an identity of the detected user; a projection controller in communication with the presence detector to receive the determined identity of the detected user; and an image projector in communication with the projection controller, wherein the projection controller transmits one or more instructions to the image projector to display an interactive image onto the surface within a vicinity of the vehicle based at least on the identity of the detected user.”
A Driver (or passenger) using gestures to interact with the heads-up display on a windshield or passenger window was further elaborated in a 2021 granted patent that we covered here.
Virtual Paper in AR/VR Environments
Patently Apple covered this invention in a May 2022 report titled “Apple has Won a Patent relating to Virtual Paper that will be used in Augmented and Virtual Reality Environments.” Apple’s patent relates to the future use of Virtual Paper on devices that will be able to crumple, flip, fold, bend while keeping the graphics intact in varying positions as the virtual paper changes angles and perspectives. Virtual paper could include mixed reality content in 2D, 3D and animation where the image could be constantly shifting depending on a particular state that it's in. This is pretty cool stuff that's now in-the-works.
Patent FIG. 4A below the 2D text "Matrix" (#410) appears to be floating on the surface of the front side of the virtual paper (#405), and the 3D chicken (#420) appears to be inside the virtual paper.
As such, the 2D text "Matrix" appears to be in front of (e.g., overlaid on) the 3D chicken from the perspective of the user. Accordingly, the 2D text "Matrix" occludes part of the crest of the 3D chicken inside the virtual paper.
Apple's patent FIG. 4B illustrates the backside of the virtual paper with the chicken, the bounded surface (#405) appears to be translucent or semitransparent, such that we can see the outline of a blurred chicken image with just the tip of the beak present on the far right of the image to show you that graphics are following the movement of the virtual paper.
The patent then goes 10 miles deep into the lighting, rasterized images, spheres, pixel depths that could lose the average reader.
As shown in patent FIGS. 4C-4E above, during the transformation of the display of the virtual paper, the display of the virtual contents associated with the virtual paper also transforms in response to transforming the contour of the virtual paper. For example, parts of the 3D-chicken protrude from one side of the virtual paper as the virtual paper or virtual page is turning.
Today’s patent update adds 20 new patent claims to better protect the patent from competitors and patent trolls. Below are just 2 of the 20 new claims:
New Patent Claim #1: “A method comprising: at a device including one or more processors, a non-transitory memory, and one or more displays: displaying a bounded region based on a first set of world coordinates, wherein content within the bounded region includes a stereoscopic virtual object displayed in accordance with a second set of world coordinates; receiving an input directed to the content; and moving the content within a perimeter of a first side of the bounded region in accordance with the input, including: moving the stereoscopic virtual object within the perimeter of the first side of the bounded region in accordance with the input; and animating the stereoscopic virtual object in accordance with the second set of world coordinates.”
New Patent Claim #12: “A device comprising: one or more processors; a non-transitory memory; one or more displays; and one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to: display a bounded region based on a first set of world coordinates, wherein content within the bounded region includes a stereoscopic virtual object displayed in accordance with a second set of world coordinates; receive an input directed to the content; and move the content within a perimeter of a first side of the bounded region in accordance with the input, including: moving the stereoscopic virtual object within the perimeter of the first side of the bounded region in accordance with the input; and animating the stereoscopic virtual object in accordance with the second set of world coordinates.”
You could review our May 2022 granted patent report for more details and graphics here and/or review Apple’s updated patent filing 20220245913 to review the full patent and the remaining 18 new patent claims here.