Today the US Patent & Trademark Office published a pair of patent applications from Apple relating to their Head Mounted Display Device project. The first invention covers an audio-based feedback system. Multiple microphones distributed across the head-mountable device will provide the user with directional audio detection. Apple's second invention covers a system that could shake loose floating particle of dust in the lens system that are interfering with the user's view of VR content and/or real-world events.
In Apple's patent background they note that a head-mountable device (HMD) can be worn by a user to display visual information within the field-of-view of the user. The head-mountable device can be used as a virtual reality (VR) system, an augmented reality (AR) system, and/or a mixed reality (MR) system.
A user may observe outputs provided by the head-mountable device, such as visual information provided on a display. The display can optionally allow a user to observe an environment outside of the head-mountable device.
Other outputs provided by the head-mountable device can include speaker output and/or haptic feedback. A user may further interact with the head-mountable device by providing inputs for processing by one or more components of the head-mountable device. For example, the user can provide tactile inputs, voice commands, and other inputs while the device is mounted to the user's head.
Head-mountable devices, such as head-mountable displays, headsets, visors, smartglasses, head-up display, etc., can perform a range of functions that are managed by the components (e.g., sensors, circuitry, and other hardware) included with the wearable device.
A head-mountable device can capture various types of inputs (e.g., visual, audio, tactile, etc.) from an environment and/or the user. A head-mountable device can also provide various types of outputs (e.g., visual, audio, tactile, etc.) to a user and/or the environment.
In particular, a head-mountable device can be provided with multiple microphones for capturing audio information (e.g., sounds) from multiple sources that are located in different directions with respect to the head-mountable device. Multiple microphones distributed across the head-mountable device can provide directional audio detection.
The head-mountable device can use the data collected by the microphones to provide visual and/or audio outputs to the user. For example, the detected audio inputs can be rendered with visual outputs by providing indicators directing the user to the source of the sound. This can allow the user to correctly and readily identify the location of the source, even when the user is not readily able to hear the sound independently of the head-mountable device.
By further example, the detected audio inputs can be rendered with audio outputs that emphasize (e.g., amplify) certain sounds over others to help the user distinguish between different sounds.
The head-mountable device can also include a speaker for audio output and/or a display for visual output. The head-mountable device can be configured to provide visual outputs based on audio inputs by displaying an indicator on a display based on a location of a source of a sound.
The head-mountable device can be configured to audio outputs based on audio inputs by modifying an audio output of the speaker based on a detected sound and a target characteristic. Such characteristics can be based on a direction of a gaze of the user, as detected by an eye sensor.
Apple's patent FIG. 1 below illustrates a perspective view of a head-mountable device on a user with multiple microphones and cameras; FIG. 3 illustrates a top view of a user wearing a head-mountable device and a source of a sound within a field-of-view of the user. Apple's patent FIG. 9 above illustrates a method of operating a head-mountable device to provide audio-based feedback with a speaker of the head-mountable device.
Apple adds that an array of multiple microphones can be operated to isolate a sound source and reject ambient noise and reverberation.
The HMD can also include one or more speakers. Where multiple speakers are provided, the speakers can be directed to each of a user's ears to provide stereo sound. Other speaker arrangements are contemplated, including surround sound.
Additionally, or alternatively, the head-mountable device can be operably connected to speakers that are directed to, near, or in a user's ears.
Various Features beyond Capturing Audio
Apple's patent provides a wide overview of a future HMD beyond the patent's initial focus on audio. For instance, Apple notes that a pair of optical modules can be provided, where each optical module is movably positioned to be within the field-of-view of each of a user's two eyes. Each optical module can be adjusted to align with a corresponding eye of the user. For example, each optical module can be moved along one or more axes until a center of each optical module is aligned with a center of the corresponding eye.
The display of the HMD is able to accurately reproduce, simulate, or augment a view based on a view captured by the camera (#150).
The HMD could also include one or more other sensors. Such sensors can be configured to sense substantially any type of characteristic such as, but not limited to, images, pressure, light, touch, force, temperature, position, motion, and so on. For example, the sensor can be a photodetector, a temperature sensor, a light or optical sensor, an atmospheric pressure sensor, a humidity sensor, a magnet, a gyroscope, an accelerometer, a chemical sensor, an ozone sensor, a particulate count sensor, and so on.
By further example, the sensor can be a bio-sensor for tracking biometric characteristics, such as health and activity metrics.
Other user sensors can perform facial feature detection, facial movement detection, facial recognition, eye tracking, user mood detection, user emotion detection, voice detection, etc.
For finer details, review Apple's patent application number 20210055367.
Particle Control for an HMD
A second HMD patent application from was published today by USPTO that generally relates to particle control for HMDs.
Apple notes that components of a head-mountable device can include one or more optical modules that provide images and/or views (e.g., to an external environment). Such optical modules can include a display element and/or optical elements that enhance a user's view.
The quality of the view provided by the optical module can be dependent on the clarity of the optical pathway between the source of the image and the eye of the user. For example, particles (e.g., dust, debris, foreign object, and/or other materials) along the optical pathway can obstruct, distort, and/or otherwise adversely affect the view provided to the user.
In particular, where a display element includes a high-resolution display, such particles on a surface of the display element can block entire pixels and/or multiple pixels, such that the image transmitted to the user is altered from its intended form.
Particles can collect along the optical pathway during the lifespan of the head-mountable device. An optical module can be sealed to prevent and/or limit ingress of substantially all particles from an external environment. However, even when an internal chamber of an optical module is sealed, particles can migrate from within the chamber due to degradation, erosion, friction, wear, and/or aging of components. When such particles are generated and/or released, they may collect on the display element and/or another optical element along the optical pathway.
Systems of the present disclosure can provide a head-mountable device with an optical module that removes particles from an optical pathway and captures the particles so they do not interfere with the user's view of and/or through optical elements.
For example, the display element and/or another optical element can be moved in a manner that releases particles on a viewing surface.
Apple's patent FIG. 1 illustrates an HMD; FIG. 3 illustrates a sectional view of an optical module of the head-mountable device.
While the optical module #140 of FIG. 3 can be sealed to isolate internal components from an external environment, it will be understood that particles may nonetheless be generated from within the housing (#144) during use and/or aging of the optical module.
For example, even when an internal chamber of an optical module is sealed, particles can migrate within the chamber from degradation, erosion, friction, wear, and/or aging of components, including components that define the chamber itself. When such particles are generated and/or released, they may collect on the display element (#170) and/or another optical element (#162) along the optical pathway (#142).
One or more actuators can be provided to move the entire optical module in a manner that releases particles that have collected on optical components.
For more details, review Apple's patent application 20210055559.
Considering that these are patent application, the timing of such devices to market is unknown at this time.