Today the US Patent & Trademark Office published a patent application from Apple relating to methods and systems behind a new handheld range finding controller for Apple's future mixed reality headset.

Head-mounted displays (HMDs), for example helmets, goggles, or glasses, may be used in a variety of applications including but not limited to virtual reality (VR) and mixed reality (MR) systems. Virtual reality (VR) allows users to experience and/or interact with an immersive artificial environment, such that the user feels as if they were physically in that environment. For example, virtual reality systems may display stereoscopic scenes to users in order to create an illusion of depth, and a computer may adjust the scene content in real-time to provide the illusion of the user moving within the scene. When the user views images through a virtual reality system, the user may thus feel as if they are moving within the scenes from a first-person point of view.

HMD with Handheld Range Finding Controller

Apple's invention covers methods and an apparatus for range finding in virtual reality (VR) and mixed reality (MR) systems. Embodiments of range finding methods and apparatus may, for example, be implemented by VR/MR systems that include a head-mounted display (HMD) such as a helmet, goggles, or glasses, and an accessory that the user is holding or wearing (e.g., a hand-held device such as a controller for the HMD or a mobile multipurpose device).

In embodiments, range finding sensors (e.g., transmitters and receivers (or transceivers), for example ultrasonic transducers), may be included on the HMD and on the accessory and used to track distances to and relative position of walls, objects, and other obstacles within constrained physical environments such as rooms, gyms, yards, or fields, or in unconstrained physical environments.

Range finding information from the sensors on the HMD and accessory can be used to generate a 3D map of the user's environment that can be used for various purposes in the VR/MR system. In addition to mapping the user's environment, embodiments of the range finding methods and apparatus may also be used to track the relative position of the accessory with respect to the HMD.

To track distances to and relative position of walls, objects, and other obstacles within physical environments, the ultrasonic transducers on the HMD and/or the ultrasonic transducers on the accessory send ultrasonic signals or "pings" that reflect or echo off surfaces in the environment; the echoes are received by some number of ultrasonic transducers on the HMD and/or on the accessory.

Time delays for the echoes can be measured and analyzed to estimate the distances and relative positions of the surfaces. Including range finding sensors such as ultrasonic transducers on the accessory allows more of the environment to be scanned than can be scanned by the range finding sensors on the HMD.

The position of the accessory with respect to the HMD can be tracked using the ultrasonic transducers, or a combination of the ultrasonic transducers with motion and orientation information obtained from an inertial-measurement unit (IMU) in the accessory and/or in the HMD.

In some embodiments, to track the accessory, the ultrasonic transducers on the HMD (e.g., three transducers) send ultrasonic signals or "pings" that are received by some number of ultrasonic transducers on the accessory. Alternatively, the ultrasonic transducers on the accessory (e.g., three transducers) send ultrasonic signals that are received by some number of ultrasonic transducers on the HMD. Time of flight for the signals from the transmitters to the receivers may be analyzed to determine distance, relative position, and orientation of the accessory with respect to the HMD.

Apple's patent FIG. 1 below illustrates an example virtual reality (VR) or mixed reality (MR) system that includes a head-mounted display (HMD) and accessory that include ultrasonic sensors for mapping the user's environment.

Apple's patent FIG. 2 below illustrates an example VR/MR system that includes ultrasonic sensors on the HMD and accessory that are used to track relative position of the accessory to the HMD.

Apple's patent FIG. 5 below illustrates mapping a real-world environment using ultrasonics in an example VR/MR system; FIG. 9 is a high-level flowchart of a method for detecting potential collisions in a VR/MR system as illustrated in patent FIG. 1.

Apple notes that while various types of range finding sensors may be used in embodiments, including but not limited to light-based sensors such as Light Detection and Ranging (LIDAR) technology sensors, embodiments are generally described that use ultrasonic transducers (e.g., ultrasonic transmitters and ultrasonic receivers, or ultrasonic transceivers) that use sound for range finding.

An advantage of using sound over light is that light-based sensors may have difficulty with some surfaces such as glass and mirrors that ultrasonic transducers can easily detect.

In addition, light-based sensors may have problems with light interference in some conditions such as bright sunlight. In some embodiments, a VR/MR system may include both light-based sensors and ultrasonic transceivers, and the range finding data from the two systems may be combined to more accurately map the user's environment.

Including range finding sensors such as ultrasonic transducers on the accessory allows more of the environment to be scanned than can be scanned by the range finding sensors on the HMD. The user holds the accessory in their hand or wears it on their arm, and thus the accessory may scan portions of the environment that are not within the range of forward-facing sensors of the HMD, for example to the side or rear of the user.

Also, the user may move their arm about while holding to the accessory to map portions of the room that are not within the range of the sensors on the HMD. In addition, the range finding sensors on the accessory be used to detect potential collisions of the user's hand or arm with surfaces and objects in the environment when the user is moving their arm; the user may be alerted to prevent the collision, for example using audio, visual, and/or haptic feedback.

Apple's patent application that was published today by the U.S. Patent Office was originally filed back in Q3 2018. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Inventors of this invention include: Wesley Zuber, Touch and Sensing Incubation Engineer; Thayne Miller, Sensing Hardware Engineer Manager; and Jae Hwang Lee, Product Design Manager.

About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.