New HMD patents this week covered Apple’s amazing Dynamic Scale for Vector Graphic Rendering, 3D Environments & more
On Thursday the U.S. Patent Office published a series of HMD patent applications relating to Vision Pro’s Dynamic Scale for Vector Graphic Rendering, adaptable HMD fit mechanisms, Optical Assembly Tracking and 3D Environments.
Dynamic Scale for Vector Graphic Rendering
Existing rendering techniques may use an undesirable amount of an electronic device's resources (e.g., a computer processing unit (CPU) and/or a graphical processing unit (GPU) computation, time, power, etc.). Existing rendering techniques may not accurately or efficiently depict complex vector graphics (e.g., text) legible, no matter the viewpoint. Further, existing rendering techniques may not provide perspective correct graphics for content imported or otherwise provided from existing applications and/or different environments.
For example, two-dimensional (2D) content may not be displayed with accurate perspective when imported into and thus viewed within a rendered 3D environment. In this way, there is a need for improved techniques for rendering text and/or user interfaces at different scales based on a distance to the user, while ensuring uniformity of scale for each text/UI object.
During Apple’s WWDC23 Keynote introducing Vision Pro, they made this point clear at the 1:30;27 minute mark which is presented below where Apple notes that “small text is crisp and easy to read at any angle.”
This week an Apple patent application covers this very aspect and advancement that part of Vision Pro and more.
Apple’s invention covers devices, systems, and methods that provide a view of a three-dimensional (3D) environment by rasterizing vector graphics (e.g., text) on separated two-dimensional (2D) assets within the 3D environment and then rendering a view of the 3D environment. The separated 2D assets may be user interface windows, buttons, etc., and may be defined as spatially-separated layers in a hierarchical drawing commands framework. The vector graphics may be rasterized to account for human vision fall-off from gaze direction to avoid over-sampling and under-sampling by using scale factors that are based on target resolutions for each spatially-separated layer.
In some implementations, the scale factor may be determined based not just on target resolutions but based on a ratio between “point units” of a layer itself and a target resolution. For example, a web browser window may be 800×800 points in size and this is invariant regardless of its distance and orientation to the user. However, once this undergoes 3D perspective projection, this will map to a varying number of pixels in the target resolution.
In some implementations, the scale factor may be variable within a single layer (e.g., user interface window). Although some implementations may select a maximum computed scale factor within a single layer, varying scale factors may be utilized within each layer by breaking the layer into zones, rather than having to select one scale for the entire layer.
In some implementations, the scale factors are determined based on vector graphic drawing commands, expected eye resolution fall-off based on gaze direction, and/or a viewer position (e.g., a six degrees of freedom (6-DoF) pose) relative to the 2D assets. In some implementations, the expected eye resolution fall-off may be determined based on a variable rate rasterization (VRR) map. With the addition of VRR, the target resolution may become variable based on which zone of the display the pixels are in. Additionally, the VRR map values may be derived based on human perception of resolution across a visual field.
In some implementations, the rasterization process may be separated from other applications. For example, the rasterization process may be provided by a separate server process on a display device so that gaze information (e.g., via the VRR map) is not provided to the application (e.g., a third-party application), and therefore, protecting user privacy by not sharing the user gaze information.
Some implementations assess physiological data (e.g., gaze information) and other user information to help improve a user experience. In such processes, user preferences and privacy should be respected, as examples, by ensuring the user understands and consents to the use of user data, understands what types of user data are used, has control over the collection and use of user data and limiting distribution of user data, for example, by ensuring that user data is processed locally on the user's device. Users should have the option to opt in or out with respect to whether their user data is obtained or used or to otherwise turn on and off any features that obtain or use user information. Moreover, each user should have the ability to access and otherwise find out anything that the system has collected or determined about him or her.
In some aspects, the asset corresponds to a spatially-separated layer of multiple spatially-separated layers of virtual content of the 3D environment, and the scale factor is a single scale factor that is applied to rasterize all vector graphics within the spatially-separated layer. In some aspects, the spatially-separated layer is a free floating window in the 3D environment. In some aspects, the asset corresponds to multiple perceptual resolution zones and the scale factor is selected to correspond to a highest resolution of the multiple perceptual resolution zones.
In some aspects, the fall-off data includes a two-dimensional (2D) mapping identifying zones around a center position, wherein the center position is determined based on the gaze direction. In some aspects, determining the scale factor for the rasterizing the vector graphic includes projecting the 2D mapping into the 3D environment, identifying a zone of the zones that corresponds with the asset in the 3D environment, and selecting the scale factor based on the identified zone. In some aspects, selecting the scale factor is further based on, determining a target resolution for each spatially-separated layer of multiple spatially-separated layers, and determining a ratio between point units of a selected spatially-separated layer and a corresponding target resolution of the selected spatially-separated layer.
In some aspects, the scale factor is selected based on the identified zone and a distance of the asset from the viewpoint position. In some aspects, the scale factor is determined to be a relatively higher scale factor when the distance is less than a threshold and determined to be a relatively lower scale factor when the distance is greater than the threshold.
In some aspects, the method is performed via a process separate from an application that provides the drawing commands, wherein the fall-off data is not provided to the application. In some aspects, the viewer position data corresponding to the viewpoint position includes a pose of the device or a head of a user wearing the device. In some aspects, the viewer position data corresponding to the viewpoint position includes six degrees of freedom (6DOF) position data.
In some aspects, the fall-off data includes a variable rate rasterization (VRR) map. In some aspects, the 3D environment includes an extended reality (XR) environment. In some aspects, the device is a head mounted device (HMD).
Apple’s patent FIG. 2 below illustrates a view #200 of an XR environment #205 provided by an HMD like Vision Pro. Additionally, FIG. 2 includes exemplary user interface's #210, #220, #230, of one or more applications (e.g., an immersive display of three window applications). Providing such a view #200 may involve determining 3D attributes of the physical environment and positioning virtual content, e.g., user interfaces (210, 220, and/or 230) in a 3D coordinate system corresponding to that physical environment.
Apple’s patent FIG. 3 further illustrates determining a scale factor for the rasterizing of the vector graphics for each 2D asset (e.g., for each user interface 210, 220, 230). The scale factor may be determined based on the fall-off data and a viewpoint position of the user.
In some implementations, based on at least a portion of the right window (e.g., representation #330 for the user interface #230) corresponding to the center zone of the VRR map (e.g., focal zone #352 of the gaze direction visualization map #350) and its distance from the viewer position, a high resolution (4×) scale factor may be selected for the user interface 230 (e.g., the right window).
Apple’s patent FIG. 4 above illustrates rendering a view of content being displayed for the XR environment of FIG. 2 based on a scale factor in accordance with some implementations. In particular, FIG. 4 illustrates using scale factors (e.g., based on target resolutions) that are determined based on (a) vector graphic drawing commands, (b) expected eye resolution fall-off based on gaze direction (e.g., a VRR map), and (c) viewer position (e.g., 6DOF pose) relative to the 2D assets (e.g., user interface 210, 220, 230).
For example, the gaze direction and the expected eye resolution fall-off based on gaze direction is illustrated by the gaze view box #410, which predicts the expected eye resolution fall-off zones based on a gaze direction detected by the device for the user. In this way, as illustrated in FIG. 4, the gaze direction focal point of the user is detected at the focus area #420, which is blown up and illustrated in area #430 (e.g., the focus of the user as they’re reading text displayed on the user interface #230).
Apple’s patent FIG. 5 is a flowchart illustrating a method for rendering a view of a three-dimensional (3D) environment based on a scale factor and a viewpoint position; IG. 7 is a block diagram of an exemplary head-mounted device.
For full details, review Apple's patent application 20240377884.
Other HMD Patents Published on Thursday Nov. 14
- 20240372328: Head-Mountable Device wit Adaptable Fit. This is a trend that we’ve covered in two previous patent reports (01 and 02) with each providing a different solution on the matter.
- 20240377647: Optical Assembly Tracking
- 20240379102: Providing and Controlling Immersive 3D Environments