Last week the US Patent & Trademark Office published two patent applications from Apple that are aimed at new developer techniques and methods relate to (1) view-based breakpoints that may offer significant advantages over traditional debugging tools for simulated reality applications, and (2) recovery techniques that can be used with foveated content, such as dynamically foveated content on future Macs and iDevices.

View-Based Breakpoints for a Display System

Head-mounted displays are used to provide virtual reality, augmented reality, and/or simulated reality experiences for users. Debugging tools are used by software developers to track down and identify errors--often called "bugs"--in computer instruction code. Debugging tools allow developers to specify breakpoints for a piece of software code, usually by specifying a line of the code as being associated with a breakpoint.

When a condition for the breakpoint occurs (e.g., a specified line of code is reached during execution of the software), then the debugging tool may trigger an action, such as pausing execution of the software at a specified line of code and/or presenting a software developer with a debugging interface that facilitates the examination of the paused state of the software and/or the controlled execution (e.g., stepping through the code one line at a time with pauses between lines) of the software.

Apple's invention covers view-based breakpoints that may offer significant advantages over traditional debugging tools for simulated reality applications.

With traditional breakpoints, this debugging task may be significantly more difficult, because, for example, the developer may need to keep their head still when creating a breakpoint to ensure that the code is running from the correct vantage point that is of interest.

With view-based breakpoints, application development time may be reduced, and/or the failure rate of resulting simulated reality applications may be reduced.

Apple's patent FIG. 2 below is an illustration of an example of a view-based breakpoint usage scenario in a virtual reality space; FIG. 3 is a block diagram of an example of a system configured to enable view-based breakpoints in a simulated reality application.

Apple's patent FIG. 6 below is a flowchart of an example of a technique for enabling a view-based breakpoint.

To review the details of this patent application for developers, see Apple's patent number 20210286701.   

Recovery from Eye-Tracking Loss in Foveated Displays

Apple's patent application relates to recovery techniques that can be used with foveated content, such as dynamically foveated content.

It should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.

Foveation refers to a technique in which some aspect of an image (e.g., an amount of detail, resolution, image quality, or brightness) is varied across displayed content based on a fixation point, such as a point or area within the content itself, a point or region of the content on which one or more eyes of a user are focused, or movement of the one or more eyes of the user.

In static foveation, various areas of an electronic display having different resolutions each have a fixed size and location on the electronic display for each frame of content displayed to the user. In dynamic foveation, the various areas at different resolutions may change between two or more images based on the gaze of the viewer.

Apple's invention presents techniques to reduce an occurrence of the reduction in image quality of the image on the display. Specifically, embodiments presented provide techniques for foveation of a display when eye tracking is not available or when an error in eye tracking occurs.

Apple's patent FIG. 2 below illustrates an eye-tracking technique that could be accomplished on a future MacBook. In patent FIG. 5 we see that it could also apply to a future iMac. Apple's patent FIG. 7B, placed within FIG. 5, is a diagram of a display in which dynamic foveation is utilized.

Apple's patent FIG. 13 abo e is a flow chart depicting operations to recover from loss of eye tracking; and FIGS. 14A-14C illustrate foveation of a portion of a display.

For developers wanting to review Apple's patent application 20210287633, click here.