Apple's latest headset inventions that came to light this week illustrated how Apple is taking their AR/VR headset project to new mind boggling heights that includes working with documents in a 3D environment using a mobile desktop replacement headset system. Patent Apple posted two key reports on their project to date (one and two) and today's third part covers methods and systems that support the headset's advanced predictive and foveated display.  

Patently Apple discovered Apple's initial patent on the subject of foveated displays in a European patent filing in late February covering an 8K Foveated Micro-Display system. The timing of the patents is interesting in light of a recent report about Apple's secret Micro-LED display plant in California.

The single inventor listed on Apple's foveated display patent filing this week is Alexander Shpunt, the former PrimeSense Chief Technical Officer who is simply now listed as Apple's 'Architect.' The PrimeSense engineering team at Apple's Israeli facility assisted in the design of Apple's TrueDepth Camera for iPhone X, so you have to take their work very seriously.

In this week's patent on their new display technology aimed at a possible future mixed reality headset Apple begins with a quick overview of today's current VR headsets.

Virtual Reality (VR) allows users to experience and/or interact with an immersive artificial environment, such that the user feels as if they were physically in that environment. For example, virtual reality systems may display stereoscopic scenes to users in order to create an illusion of depth, and a computer may adjust the scene content in real-time to provide the illusion of the user moving within the scene. When the user views images through a virtual reality system, the user may thus feel as if they are moving within the scenes from a first-person point of view.

Similarly, augmented reality (AR) combines computer generated information with real world images to augment, or add content to, a user's view of the world. The simulated environments of virtual reality and/or the enhanced content of augmented reality may thus be utilized to provide an interactive user experience for multiple applications, such as interacting with virtual training environments, gaming, remotely controlling devices or other mechanical systems, viewing digital media content, interacting with the internet, or the like.

However, conventional virtual reality and augmented reality systems may also suffer from latency problems potentially cause eyestrain, headaches, and/or nausea. For example, conventional virtual reality and augmented reality systems may involve significant lag times between the time a user looks in a particular direction and the time that the system is able to display the corresponding scene to the user.

Additionally, the amount of image data required to be captured, generated and/or displayed to the user of a conventional virtual reality system may be so large that it affects the performance of the system (e.g., increased latency).

Apple's Predictive, Foveated Virtual Reality System

Apple's invention covers methods and systems for a virtual reality (VR) and/or augmented reality (AR) device (e.g., a headset, or head mounted, device) that may include an AR/VR display system that is both predictive and foveated. A predictive, foveated virtual reality system may be configured to capture views of the world around a user of the system, augment the captured data, generate an augmented view of the world and display that view to the user via a display of the system.

A predictive foveated virtual reality system as described in this patent filing may also be configured to capture (and render) image data using more than one resolution, such as to possibly reduce overall overhead, cost, workload. For instance, a virtual reality system may include one or more cameras (or camera systems) configured to capture both low resolution and high resolution image data. Low (or lower) resolution image data may be captured for a wide field of view (e.g., the periphery) while high (or higher) resolution image data may be captured for a narrow field of view (e.g., corresponding to a user's line of sight). Thus, in some embodiments, a predictive, foveated virtual reality system may include one or more lower-resolution cameras in addition to one or more higher- resolution cameras.

Additionally, a virtual reality system may be configured to anticipate a user's movements, such as to look ahead, in order to capture image data, as well as prepare the image data for display, for a predictive, future line of sight, according to some embodiments. For example, a virtual reality system may be able to determine, based on a user's head and/or eye motion, a predictive line of sight vector along which to capture image data. When the user actually looks in that direction (e.g., the user's actual line of sight approaches the predictive line of sight vector), the system may then display the previously captured (and augmented) view.

Additionally, a predictive, foveated virtual reality system may include one or more sensors and/or mechanisms configured to detect, determine and/or track motion. For example, a predictive, foveated virtual reality system may include one or more gaze tracking modules configured to detect, determine and/or track eye motion for a user of the system. Similarly, a predictive, foveated virtual reality system may include one or more accelerometers, gyroscopes, and/or other components, configured to detect, determine and/or track general motion of the system and/or of a user's head while wearing/using the system.

Apple's patent FIG. 1 presented below is a logical diagram illustrating part of a system configured to implement a Predictive, Foveated Virtual Reality System; FIG. 2 is a flowchart illustrating one embodiment of a method for implementing a Predictive, Foveated Virtual reality system.

Apple's patent FIG. 3 below is a logical diagram illustrating one embodiment of Predictive, Foveated Virtual Reality System configured to capture image data corresponding to user's line of sight.

Apple's patent FIGS. 10 and 11 are logical diagrams illustrating embodiments of a camera configured to capture image data over a range of view angles.

Apple's patent application # 20180081178 was filed back in Q3 2017. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.