Apple Smartglasses Patent Reveals Gaze-Driven Recording of Video and AR Content with Jitter Reduction
Today the US Patent & Trademark Office published a patent application from Apple that relates to their future smartglasses device that could record content including AR content with jitter reduction and more.
Apple notes that head-mounted displays are used to provide virtual reality, augmented reality, and/or mixed reality experiences for users. Video from a virtual reality or mixed reality experience may be recorded for later playback or analysis. Gaze-tracking sensors may be used to provide an indication of where a person is currently looking. Instead of a camera on the smartglasses frame simply recording what's in front of the user, the camera could be driven by where the user's eyes or gaze is shifting to.
Apple's invention covers systems and methods that provide gaze-driven signal processing to facilitate mixed reality or virtual reality applications, such as mixed reality recording.
In mixed reality recording, a portion of what the user sees may be recorded. This portion of what the user sees (e.g., including frames of video presented to the user via a head-mounted display and/or frames of video of a surrounding environment that are captured using one or more image sensors mounted on a head-mounted display) may be referred to as a recording box.
The recording box can be a fixed area in relation to the video input, but then the user might be looking at interesting content outside of this region that the recording will miss. Systems proposed herein identify the region of interest by tracking the user's gaze and move the recording box as the user's gaze moves.
One such example can be that the gaze is in the center of the recording box to the extent that the recording box is contained in the original frame. The recording box can be a wide variety shapes, including for example, oval, hexagon, or rectangle, among others.
Moving the recording box with the gaze center point can result in a jittery video. Some implementations run a smoothing or stabilization algorithm on top of gaze center points to avoid jitter in recorded video and get a more pleasant recording.
This idea has use cases beyond mixed reality recording, such as running feature extraction in a region of interest; and variable bit allocation, higher in a region of interest and lower in surrounding areas.
Apple's patent FIG. 4 below is a block diagram of an example of a system configured to enable gaze-driven processing of video. The system consists of a pair of smartglasses and connected device for power and storage such as an iPhone, iPad or MacBook.
Apple's patent FIG. 1B above is an illustration of an example of a process for recording video in mixed reality or virtual reality applications with a gaze-driven region of interest; FIG. 1C is an illustration of an example of a process for recording video in mixed reality or virtual reality applications with a smoothed gaze-driven region of interest.
Apple's patent FIG. 6 below is a flowchart of an example of a process for gaze-driven processing of video.
Apple notes that smartglasses will be able to work with an iPhone, an iPad, a MacBook, a desktop computer, a smart TV, a smart speaker and more.
Apple's patent application 20200195940 that was published today by the U.S. Patent Office was filed back in Q4 2019. Considering that this is a patent application, the timing of such a product to market is unknown at this time. For those wanting to drill into the details of this invention can do so here.