It's unknown at this time if the drastic drop in Apple's patent applications and granted patents over the last two months is reflecting an issue with the U.S. Patent Office or reflecting the effect of COVID-19 had on Apple's ability to process patents during those years. Today, for instance, only 27 patent applications were published in contrast to an average of 70-150 in the past. That's a dramatic drop.

With that said, today the US Patent & Trademark Office published a patent application from Apple that relates to systems, methods, and devices that detect surfaces that provide specular reflections, modify reflections, or provide content that depicts such surfaces and reflections. More specifically, Apple's patent describes how a user wearing an HMD could be holding an iPhone in the real-world but have it transform into another device like a light saber in a game. Other examples are provided for in the patent. This is a patent that perhaps Apple developers will better appreciate.

According to Apple, various implementations of this invention include devices, systems, and methods that detect mirrors, glass, liquids, and other surfaces and reflections in such surfaces. Some implementations involve detecting a reflection of a user or the user's device in a surface and providing virtual content that enhances or replaces the appearance of the reflection in content provided to the user.

For example, a view of an environment may be modified to include a reflection of a user that does not include the electronic device that the user is holding or wearing.

In another example, an environment may be modified so that virtual content, such as a newer version of the electronic device or a virtual light saber, replaces the electronic device in a reflection.

In another example, the environment may be modified so that virtual content, such as an avatar, replaces the user in a reflection.

The method may further involve identifying replacement content to replace at least a portion of the reflection of the user or the electronic device. In some implementations, the replacement content includes portions of the user's body (e.g., a hand, an arm, etc.) or background content that will be visible when a handheld mobile device is erased, enhanced, or otherwise modified.

In some implementations, the replacement content includes portions of the user's body (e.g., an arm, clothing, etc.) or background content that will be visible when a wrist-mounted device is erased, enhanced, or otherwise modified.

In some implementations, the replacement content includes portions of the user's head (e.g., cheeks, eyes, eyebrows, hair, etc.) or background content that will be visible when a head-mounted device (HMD) is erased, enhanced, or otherwise modified.

In Apple's patent FIGS. 4 and 5 below we see how Apple will allow a device held by a user in the real-world to transform into another device, like a light saber in the CGR environment. In FIG.5, although in the real-world the user is wearing an HMD, in a CGR environment the HMD is replaced with the user's face without an HMD.

Expressions of the user are captured by the cameras that are situated inside the headset. Apple further notes: "For example, expressions of the brow, mouth, jaw, and eyes captured by sensors (e.g. cameras) may be used to simulate expressions on an avatar of the user.

Apple's patent FIG. 6 above illustrates a system flow of HMD device #120 detecting surfaces providing specular reflections and providing a CGR environment. The HMD device uses one or more environment sensors #610 (e.g., one or more cameras) to capture, create, or provide a view #130 of a physical environment #100.

Elsewhere in the patent, Apple notes that "the user may view and otherwise experience a CGR-based user interface that allows the user to select, place, move, and otherwise experience and interact with a CGR environment, for example, via hand gestures, voice commands, input device inputs, etc."

Lastly, Apple notes that projection-based systems may employ retinal projection technology that projects graphical images onto a person's retina. Projection systems also may be configured to project virtual objects into the physical environment, for example, as a hologram or on a physical surface.

For those wanting to go deeper into Apple's Rabbit Hole, check out Apple's patent application number 20230131109.

Apple's Inventors

• Selim Ben Himane:  Director Computer Vision & Machine Learning in the field of AR and VR.
• Brian Christopher Clark: Engineering Program Manager, Technology Development Group
• Daniel Kurz: Senior Machine Learning Manager (Computer Vision, Machine Learning, Augmented Reality)
• Peter Georg Meier: was Apple's algorithm Lead until 2019. He was the former owner of Metaio GmbH that Apple acquired in 2015.