During Apple's WWDC23 introduction to their revolutionary Vision Pro Spatial Computing headset, Mike Rockwell, VP, Technology Development Group, came on stage to describe the key components behind the new device. In the segment where Rockwell introduced the new visionOS, he pointed out the OS's Foveated Renderer. He noted at the new "dynamically foveated rendering pipeline delivers maximum image quality to exactly where your eyes are looking for every single frame." 

Today the US Patent & Trademark Office published a patent application from Apple titled "Perspective Correct Vector Graphics with Foveated Rendering," that relates to devices, systems, and methods that implement rendering processes that performs vector graphic rendering based on information received from a source application. Various implementations disclosed herein include devices, systems, and methods that implement foveated rendering using content received from a source by selectively drawing the content for only some regions based on gaze.

In some implementations, the techniques disclosed in their patent filing may be implemented on a head-mounted device (HMD) having an optical see-through or opaque display.

XR systems may detect head movement and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment.

Apple's patent covers devices, systems, and methods that perform a rendering process that performs foveated text rendering using content received from a source. For example, the content that would be displayed by an app on the screen of a mobile device may be displayed using a flat virtual surface at a position within a 3D environment. The rendering process can provide foveated rendering by drawing regions based on gaze direction (e.g., drawing only some regions, drawing some regions at higher resolution/higher frame rate, etc.). In some implementations, the rendering process provides an increase in quality in a focus region and/or reduces the amount of computation and memory used in providing perspective correct rendering.

Apple's patent FIG. 1 below illustrates left eye and right eye stereoscopic images provided by an electronic device in providing an extended reality (XR) environment; As shown in FIG. 2D, the foveated rendered user interface #210 is rotated about the x-axis and the y-axis. Such rendering may occur, for example, when the virtual display surface is positioned at a relatively close up and at an angled orientation relative to the electronic device.

For finer details, review Apple's patent application 20230186538. The invention may apply to future iPhones, iPads, Macs and Apple Watch.  

Apple Inventors

• Yidi Zhu: Sr. AR Software Engineer (with a passion for game design)
• Jeremy Bridon: Software Engineer (specializing in computer graphics and mobile systems)
• Courtland Idstrom: Software Engineer (Gaming Industry background)