Patent of the Decade: Apple Reveals an Unbelievable VR Experience System for Next-Gen Autonomous Vehicles
There have been a series of very smart and very cool patent applications surfacing of late covering a future mixed reality headset that could one day be a part of a virtual desktop system designed to edit 3D documents. There was also a very detailed invention covering an advanced gesturing control system for a heads-up-display system of an autonomous vehicle.
Today the US Patent & Trademark Office published a patent application from Apple that has to be the invention of the decade. It may never see the light of day, but boy is it cool. Apple notes that "embodiments of the VR system as described herein may be implemented in autonomous or "self-driving" vehicles where all occupants are passengers.
The VR system may enable safer, smaller, and less expensive autonomous vehicles. Windows in vehicles are inherently unsafe and not structurally sound, and add cost to vehicles. By providing a virtual view of the real environment or of a simulated environment, the VR system may reduce or eliminate the need for windows in autonomous vehicles, allowing the vehicles to be engineered with fewer and/or smaller windows, or with no windows at all.
In addition, a VR experience provided through the VR system may provide passengers with the sensation that they are actually riding in a larger vehicle than the actual size of the autonomous vehicle, which may provide the passengers with a more pleasant and secure-feeling experience when riding in a small autonomous vehicle.
Perhaps just as important, Apple introduces us to their VR Experience System that could provide passengers with fun and wild entertaining VR experiences that is really creative.
Yes, it's futuristic to be sure, but this is just Apple's beginning vision and it's going to make younger Apple fans want one of these autonomous vehicles. They won't think of it as an autonomous vehicle but rather a virtual arcade on wheels.
The Immersive Virtual & Augmented Display System
Apple's patent FIG. 1 presented below illustrates a virtual reality (VR) system including a head mounted device (HMD) that may be used by passengers in vehicles, according to some embodiments; FIG. 2 illustrates a VR system that projects VR content to a window of a vehicle for viewing by passengers; and FIG. 3 illustrates projecting VR content so that it appears to the viewer to be in space in front of the vehicle.
Apple's patent FIG. 6 above illustrates a VR system in a vehicle; FIG. 7 below is a block diagram illustrating components of a VR system in a vehicle.
Apple's second patent on this invention adds augmented reality: the virtual content is displayed in a view of a real world scene in front of the vehicle to provide an augmented or mixed reality view to the passenger to include virtual objects and virtual tags.
Immersive VR Experiences
Apple's patent FIG. 8 below is a block diagram illustrating active systems in a vehicle that may be used in a VR system to provide synchronized physical effects for passengers.
Apple provides several example immersive VR experiences that may be provided by embodiments of a VR system, and are not intended to be limiting.
Embodiments of the VR system may provide immersive VR experiences to passengers in vehicles, for example by replacing the view of the real world with any of various types of virtual experiences and environments that a passenger may desire.
Vehicle motions may be integrated into the virtual experiences, for example to help prevent motion sickness and to enhance the virtual experience. Integrating the VR system with a vehicle in motion provides opportunities for enhancing virtual experiences that are not available while sitting in a room using a stationary simulator or wearing a HMD.
For example, accelerations and motions in a virtual experience can be matched to or enhanced by accelerations and motions of the vehicle. In some embodiments, active vehicle systems (e.g., HVAC systems, audio systems, and active seats) and/or vehicle control systems (e.g., braking, throttle, steering, and active suspension systems) within constraints, may be integrated with the VR system to provide physical effects with the virtual experience, for example rushing wind or heat through the HVAC system, surround sound and sound effects through the audio system, and acceleration or motion effects through the seat.
In two patent under FIG 9 show examples of immersive VR experiences that may be used to enhance productivity while riding in vehicles, while also providing an interesting VR experience for participants. For example, two or more people may hold a meeting as avatars around a table in a virtual environment, for example in a virtual room as shown in FIG. 9 (B) or on the bed of a flatbed truck.
Two patent figures show experiences of taking a canoe trip in rapids and running over zombies with your vehicle in a post-apocalyptic landscape.
Other experiences could include a wild hand gliding flight over a canyon. The virtual experiences may also be educational and interactive, for example allowing the passenger to discover history or other information about landmarks in a virtual view of a city that they are experiencing. The virtual experiences may be interactive in other ways, for example allowing a passenger to pass other vehicles during a virtual road race experience.
As shown below in patent FIG. 11, we're able to see virtual views of real or fictional people that could be integrated into the virtual experience provided by the VR system. The example below is that of a virtual representation #1195 of an author or talk show host appearing to be sitting in the seat next to the passenger #1190; the virtual author may be reading one of their books to the passenger, or the virtual talk show host may be hosting their show from the seat next to the passenger, with their voices provided through the audio system of the vehicle or the HMD #1100.
Oh, and about that crazy flatbed truck experience for a meeting. Well, you can switch that scenario out and have your favorite band giving you and your friends a private concert. You'll be able to add the car's audio system to pump up the volume.
Monitoring a Passenger
Apple's patent FIG. 18 below illustrates monitoring a passenger #1890 using a VR system in a vehicle. The virtual content #1840 and audio #1842 generated by a VR application executing on a VR controller #1810 and presented to a passenger via an HMD #1820 may be adapted to the particular passenger according to the passenger's preferences and/or according to passenger sensor data #1830 collected by sensors in the vehicle.
In some embodiments, a VR application may initialize a VR environment for the passenger according to predefined or predetermined passenger preferences noted in patent figure 19 above.
For example passenger preference data maintained in a preferences file stored on the controller or accessed from external storage such as cloud storage. In some embodiments, the VR system may provide an interface that allows the passenger to specify their preferences for a VR environment to be experienced (e.g., the passenger may request a relaxing, normal, or exciting experience).
In some embodiments, while using the HMD to experience the VR world generated by the VR application on the VR controller, various sensors may be used to monitor the passenger for signs of discomfort or motion sickness (e.g., paleness, sweating, fidgeting, swallowing, burping, pulse rate, breath rate, eye motions, etc.); the passenger sensor data #1830 may be provided to the VR controller via wired or wireless connections.
In some embodiments, the sensors may include sensors #1832 in or on the HMD, for example internal cameras that monitor the passenger's eyes or other portions of the passenger's face, external cameras that monitor other parts of the passenger's body such as the arms and hands, Inertial-measurement Units (IMUs) that detect and track motion of the passenger's head, and/or sensors that monitor physiological responses such as sweating, swallowing, and breath rate.
In some embodiments, the sensors may include sensors attached to or worn on other parts of the passenger's body, for example wrist bands, arm bands, or watches that monitor sweating, pulse rate, swallowing, or other physiological responses, and/or that include IMUs that detect and track motion of the passenger's body.
Establishing a Virtual Ground Plane
As shown in patent FIG. 15 below, a virtual ground plane or platform #1504 may be provided beneath the passenger in the virtual environment #1500, for example positioned in the virtual environment where the passenger's feet would be, to help avoid symptoms of motion sickness. This prevents the passenger from having the sense of flying through space without a ground plane beneath them, which is disconcerting to some passengers. In some embodiments, virtual representations of the passenger's arms and legs may also be displayed.
As shown in FIG. 15 above, anchored virtual content #1506 may be provided to help passengers orient themselves in the real world while experiencing a virtual environment in a moving vehicle.
In some embodiments, an object or aspect may be displayed in the virtual environment that is anchored to a real world direction (e.g., the sun or moon anchored at due west or due east, or a tall building anchored at due north) so that the passenger may more easily track where they are and how they are oriented in the real world as the vehicle follows a real-world route. As the vehicle makes a turn in the real world, the virtual world (including engaging virtual content #1502) turns with the vehicle, but the anchored virtual content remains fixed relative to the real world.
In some embodiments, preview information about upcoming driving maneuvers and real-world terrain effects that can be physically felt (e.g., right turns, left turns, stops (e.g., at red lights or stop signs), climbs in elevation, dips, speed bumps, downhill portions, etc.) may be provided for improved passenger comfort. In VR experiences where there is no video see-through of the real world and in which no cues about maneuvers are provided, passengers may experience discomfort from not being able to tell where they are in the real world, and from not being able to anticipate and brace themselves for upcoming driving maneuvers which physically affects their vestibular/balance sense.
The preview information for driving maneuvers and real-world terrain effects may be provided by visual, audio, and/or haptic cues. For example, visual cues (e.g., directional arrows) may be displayed to indicate upcoming left or right turns, climbs, or descents.
As another example, audio cues may be played to indicate upcoming maneuvers; for example, a sound or tone may be played to the left ear to indicate an upcoming left turn, and to the right ear to indicate an upcoming right turn, with different tones or sounds played to both ears to indicate upcoming descents or climbs.
Apple's patent application # was filed back in Q3 2017. Two of the inventors listed on the patent are Mr. Lyon, Director, Sensing Hardware and Mr. Kurz, Computer Vision and Machine Learning Manager.
Considering that this is a patent application, the timing of such a product to market is unknown at this time.
Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of patent application 20180089900 should be read in its entirety for full and accurate details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.