Apple Patent focuses on integrating Inertial Measurement Units into future MR Headsets & Vehicles to enhance VR Experiences
The U.S. Patent and Trademark Office officially published a series of 51 newly granted patents for Apple Inc. today. In this particular report we cover Apple's patent relating to headsets for gaming, vehicle simulators and other applications. Apple has worked on a few headset patents covering the need for accurate positional tracking and methods to avoid motion sickness while wearing a headset in a vehicle.
In March 2018 Patently Apple posted a report titled "Patent of the Decade: Apple Reveals an Unbelievable VR Experience System for Next-Gen Autonomous Vehicles." The patent was in context with a passenger in a future autonomous vehicle using a headset to both entertain and get work done.
The report noted that "Vehicle motions may be integrated into the virtual experiences, for example to help prevent motion sickness and to enhance the virtual experience. Integrating the VR system with a vehicle in motion provides opportunities for enhancing virtual experiences that are not available while sitting in a room using a stationary simulator or wearing a HMD."
In another patent report posted in June of this year, it discussed creating smoother CGI scenes for VR headsets that could eliminate Eye Strain and Nausea.
This is a trend in some of Apple's headset patents and today is no different. In fact the engineer behind this patent went down the rabbit hole and in great detail meticulously explained how playing games while riding in a vehicle could pose problems with conflicting motion between the real and virtual world environments that could lead to nausea. Today's patent is focused on headsets worn in vehicles and keeping the motion and inertia factors in balance.
Apple notes that Virtual reality (VR) allows users to experience and/or interact with an immersive artificial environment, such that the user feels as if they were physically in that environment. For example, virtual reality systems may display stereoscopic scenes to users in order to create an illusion of depth, and a computer may adjust the scene content in real-time to provide the illusion of the user moving within the scene.
Further, when the user views images through a virtual reality system, the user may feel as if they're moving within the scenes from a first-person point of view.
Similarly, augmented reality (AR) combines computer generated information with real world images to augment, or add content to, a user's view of the world.
The simulated environments of virtual reality and/or the enhanced content of augmented reality may thus be utilized to provide an interactive user experience for multiple applications, such as interacting with virtual training environments, gaming, remotely controlling drones or other mechanical systems, viewing digital media content, interacting with the internet, or the like.
In addition, VR systems and/or AR systems may utilize inertial measurements from an inertial measurement unit (IMU) included in a VR or an AR headset to determine how images are to be displayed.
Conventional virtual reality and augmented reality systems may not be able to separate motion of a user or a user's body part from motion of a reference frame in which the user is travelling, such as a vehicle in which the user is travelling.
For example, a user wearing a conventional VR or AR device may be seated in a vehicle and the vehicle may accelerate from a stopped position to a high speed while the user wearing the VR or AR device sits in the vehicle without moving within the vehicle (e.g. no relative motion of the user relative to the reference frame of the vehicle).
Because the conventional VR or AR device cannot separate the motion of the user's body from the motion of the vehicle, the conventional VR or AR device may attribute the motion of the vehicle to the user. In this way images displayed to the user on the VR or the AR headset may appear to the user as if the user is running through a scene at the same speed and in the same direction the vehicle is travelling.
A similar phenomenon occurs in regard to angular motion. For example, a user wearing a conventional VR or AR device may be riding in a vehicle that turns, however the user may not actually turn their head when the vehicle turns. A conventional AR or VR device may not be able to separate the motion of the user's head (e.g. not turning) from the motion of the vehicle (e.g. turning).
Therefore, the turning motion of the vehicle may be attributed to the user and images displayed to the user on the VR or AR headset may appear to be turning or spinning despite the user not turning their head.
Such discrepancies between a user's relative motion within a vehicle and motion observed by the user via a scene displayed to the user may lead to nausea and sickness of the user. For example, nausea may be caused by oculovestibular mismatch (see).
Likewise, in the case of other types of devices that include IMUs, such as controllers, motion of a vehicle being attributed to the controller may lead to erratic control and unintended consequences.
Apple's invention in-part covers embodiments of a system, a head mounted display and methods that implement relative inertial measurement technology to determine relative motion of a user or a part of a user's body relative to a non-fixed reference frame, such as a vehicle in which the user is travelling.
In some embodiments, relative inertial measurement technology may be used to determine images to be displayed to a user via a head mounted display based on relative movements of the user's head while the user is travelling in a non-fixed reference frame, such as a vehicle.
In some embodiments, relative inertial measurement technology may be used with a controller coupled to a user's body to determine motion of the user's body relative to motion of a non-fixed reference frame in which the user is travelling, such as a moving vehicle.
Apple's patent FIG. 1 below illustrates a user riding in a vehicle and a system including multiple inertial measurement devices for determining relative motion of the user relative to the vehicle.
Further to patent FIG. 1, Apple notes that while the user is shown to be riding in a car, the fact is that the user may be riding in various types of vehicles, such as trains, planes, subways, boats, elevators, or other types of vehicles.
The user device #104 above is a head-mounted display (HMD) may be part of an inertial measurement system, such as system #106 that determines relative motion of a part of the user's body to which the user device is coupled relative to a reference frame in which the user is riding.
Apple's patent FIGS. 2A to 2C above illustrate examples of relative motion of a user relative to a vehicle reference frame.
Apple's patent FIGS. 3A-3C below illustrate examples of relative angular motion of a user relative to a vehicle reference frame
Apple's patent FIG. 4A above illustrates a block diagram of an example inertial measurement device.
Apple's patent FIG. 5 below illustrates an example of a Mixed Reality headset configured to determine relative inertial motion.
Apple's patent FIG. 10A above is a logical block diagram of a relative inertial measurement system that determines relative inertial motion based on time synchronized inertial measurements.
Apple's granted patent 10,503,245 was originally filed in Q3 2016 and published today by the US Patent and Trademark Office.
The inventor listed on the granted patent is 13 year veteran by the name of Fletcher Rothkopf that is Senior Director of Product Design, Technology Development Group.