Last year major revelations unfolded regarding Apple's future mixed reality Head Mounted Headset (HMD) in at least 3 major patent filings (01, 02 & 03) that were mind boggling in scope covering entertainment, work, and autonomous vehicles.  This week Apple's fourth major patent covering their HMD surfaced detailing various aspects of their all-important gaze control system along with, a retinal projector, it's video processing engine, sensors covering in-air 3D hand gestures,  face gestures and more.

Apple notes that as the technology for capturing video has improved and become less expensive, more and more applications with video components are becoming become popular.  For example, mixed reality applications (applications in which real-world physical objects or views may be augmented with virtual objects or relevant supplementary information) and/or virtual reality applications (applications in which users may traverse virtual environments), in both of which video data may be captured and manipulated, are an increasing focus of development and commercialization.

For at least some applications, video data representing the environment may be processed at a device other than the video capture device itself; that is, video data may have to be transmitted over a network path (such as a wireless link) which may have relatively low bandwidth capacity relative to the rate at which raw video data is captured.

Depending on the video fidelity needs of the application, managing the flow of video data over constrained network pathways while maintaining high levels of user satisfaction with the application may present a non-trivial technical challenge.

Apple's HMD: Gaze & Pre-Filtering Video Data

Apple's invention covers various methods and apparatus, a mixed reality headset, for gaze direction-based pre-filtering of video data. In at least some embodiments,  the filtering techniques  may  take advantage of the fact that the visual acuity or precision of perception in the human visual system typically decreases as a function of the angular distance away from the central direction of the gaze - the portion of a visible scene which is sensed using a centrally-located region of the retina called the fovea is typically perceived with greater sharpness than portions of the scene sensed using  portions  of  the  retina  that  are  located  away  from  the  fovea. 

Taking  this  biological phenomenon into consideration, video data which is to be transmitted over a network may be pre- filtered  (prior  to  compression)  using  a  configurable  multi-layer  low-pass  filter  in  various embodiments,  with  outer  or  peripheral  regions  of the  visible  scene  being  filtered  at  lower resolution  settings relative to the inner or central regions relative to the direction of the gaze, thereby helping to reduce the total amount of data that has to be transmitted over the network.

A method may comprise tracking the direction of the gaze of an individual, e.g., using one or more sensors of a head-mounted device such as a headset or helmet which are directed towards the individual's eyes.

The method may include filtering one or more frames of video data representing at least a portion of an environment of the individual using a multi-layer low-pass filter. A given frame of video data which is filtered may include representations of one or more physical objects and/or virtual objects (e.g., objects generated by virtual reality or augmented reality applications) in various embodiments.

In at least one embodiment, the video processing engine may be configured to generate a modified visual representation of the environment - e.g., by augmenting the originally-viewed scene with virtual objects or with supplementary information about real objects that are visible - and transmit the modified version for viewing by the individual. As the individual interacts with the visible environment (which may comprise a mix of real and virtual objects in some embodiments, and may be referred to as a mixed-reality environment), the direction of the individual's gaze may change from one set of displayed/visible frame to another in various embodiments.

Apple's patent FIG. 1 presented below illustrates an example system environment in which video data captured at a wearable device may be pre-filtered using a multi-layer low pass filter and compressed prior to transmission to a processing engine.

A mixed reality (MR)  system  may  combine  computer generated information (referred to as virtual content) with real world images or a real world view to augment, or add content to, an individual's view of the world, or alternatively  may combine representations of real world objects with views of a computer generated three-dimensional (3D) virtual world.

In some embodiments, components of an MR application or system may, for example, include a head mounted device HMD (#102) such as a headset, helmet, goggles, or glasses that may be worn by an individual or user (#190) and a base station (#160).

The base station may comprise a processing engine (#184) configured to render mixed reality frames including virtual content (#110) for display by the HMD. The HMD and base station may each include wireless communications technology that allows the HMD and base station to communicate and exchange data via a wireless connection 180.

In some embodiments, world sensors (#140) may collect additional information about the user environment (e.g., depth information, lighting information, etc.) in addition to video. Similarly, in some embodiments, user sensors (#150) may collect additional information about the individual, such as expressions, hand gestures (a common Apple theme of late 01 & 02), face gestures, head movements, etc.

A 3D virtual view (#104 seen in FIG. 1 as a defined bubble) may comprise a three-dimensional (3D) space including virtual content  (#110) at different  depths that individual sees when using the mixed reality system of FIG. 1.

In some embodiments, in the 3D virtual view the virtual content may be overlaid on or composited in a view of the individual environment with respect to the user's current line of sight that is provided by the HMD.

Apple's HMD may implement any of various types of virtual reality projection technologies in different embodiments.   For example, the HMD may implement a near-eye VR technique that displays left and right images on screens in front of the individual eyes that are viewed by a subject, such as techniques using DLP (digital light processing), LCD (liquid crystal display) and LCoS (liquid crystal on silicon) technology VR systems.

As another example, Apple's HMD may comprise a direct retinal projector system that scans left and right images, pixel by pixel, to the subject's eyes.  To scan the images, left and right projectors may generate beams that are directed to left and right  reflective components (e.g., ellipsoid mirrors) located in front of the individual's eyes; the reflective components may reflect the beams to the eyes.

To create a three-dimensional (3D) effect, virtual content at different depths or distances in the 3D virtual view may be shifted left or right in the two images as a function of the triangulation of distance, with nearer objects shifted more than more distant objects.

While not shown in FIG. 1, in some embodiments a mixed reality system may include one or more other components. For example, Apple's HMD may include an exterior trackpad in order to assist the user interact with 3D virtual content.

Apple's patent FIG. 9 below is a block diagram of a mixed-reality system in which pre-filtering of video data may be performed.

Apple's patent FIG. 4 below illustrates examples of subsets of a video data frame to which filtering functions of a multi-layer low pass filter may be applied; FIG. 8 is a flow diagram illustrating aspects of operations which may be performed to set initial values for, and later dynamically modify, parameter settings for pre-filtering video data.

Apple's patent FIG. 2 below illustrates an overview of a workflow in which video data is pre-filtered and compressed at a wearable device prior to being analyzed.

Apple's patent application was originally filed back in Q3, 2018. Considering that this is a patent application, the timing of such a product to market is unknown at this time. As a side note, it would appear that 5G will be able to advance Apple's project tremendously and kill some of the video bottlenecks.

Although completely coincidental, this week a major Samsung patent came to light covering one of their next-gen VR headset inventions that will provide a massive curved HDM display delivering a more immersive experience that we covered on Friday here.  

About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.