Today, the U.S. Patent and Trademark Office officially granted Apple a patent that relates to head-mounted displays (HMDs) with both internal and external displays. The external display may facilitate interaction between a wearer and another person or an environment. It basically covers an advanced version of Apple Vision Pro's EyeSight feature that extends the external display's range of communicative images and symbols beyond the user's eyes.  For instance, when a user is playing a video game, the external display may present the game's title being played that communicates "don't disturb."

Wearable Device for Facilitating Enhanced Interaction

Apple's granted patent covers Apple Vision Pro's EyeSight feature that could expand in the future. Technically, Apple notes that HMDs that have an outward-facing output system (including display(s), a speaker(s), or the like) in addition to wearer-facing displays to increase the options for interacting with others while wearing the HMD. In particular, HMDs that completely cover a wearer's eyes (or even entire head), such as those with displays or projectors enclosed in an opaque housing, may provide an immersive experience for the wearer. Further, HMDs may present environments and images to the wearer that are highly engaging and immersive, so much so that the wearer may be largely oblivious to what is happening in the real-world environment. For example, not only might an HMD block a wearer's view of the real-world environment, but a virtual-reality environment presented by the HMD may consume a wearer's attention and focus, making the wearer less likely to hear people or be able to respond or react to the real-world environment.

Further, such HMDs limit the ability of people in the real-world environment to recognize or interact with the wearer, as the wearer's eyes are usually completely covered and the wearer's attention may be entirely directed to a virtual environment. For example, even if a HMD allows a wearer to see the external environment (e.g., via an external camera transmitting images to a wearer-facing display), a person who is in front of the wearer cannot determine if the wearer is able to see or perceive the real world environment, or if they are immersed in a virtual environment. And even if a person and the wearer are interacting by speaking, the wearer's eye movements and facial expressions remain largely hidden, thus removing many non-verbal cues and signals that are important for conveying information and emotion in human-to-human communications.

Accordingly, Apple's patent covers HMDs that incorporate outward-facing or outwardly-directed output systems, such as external displays (high and/or low resolution displays), speakers, or the like, that can display or otherwise present information to observers in the real-world environment. This outwardly displayed or presented information may provide a communication path that helps penetrate the otherwise immersive world of a virtual-reality environment and helps break down the physical barrier of the HMD.

For example, outward-facing displays may be capable of displaying symbolic graphical information to outside observers, and speakers may allow a digital assistant to interact with another person until a wearer is able to direct their attention to the other person. In this way, communication between the wearer and the outside observer is restored or improved, despite the immersive and dissociative effect of the HMD and the virtual environment.

As one particular example, in cases where an HMD includes an outward-facing graphical display, aspects or images of a wearer's face may be displayed to the real-world environment, thereby improving interaction, recognition, and communication between the wearer and individuals in the real-world environment.

To facilitate these improvements, an HMD may include a camera that captures images (e.g., still images or video images) of the portion of a user's face that is covered by a HMD, such as the user's eyes, eyebrows, nose, forehead, or the like. The HMD may display those captured images on the outward-facing display, allowing individuals in the real world to see the wearer's eyes and facial expressions, thereby facilitating a more natural interaction with the wearer. Moreover, the outward-facing display may also allow others to more quickly and easily recognize the person wearing the HMD, as more of their face will be perceptible. As another example, where an HMD includes a low resolution display (e.g., an LED array), the display can be used to display symbolic graphical outputs that correspond to or suggest a mood or emotion of the wearer, or that convey a mode or status of operation of the HMD.

As yet another example, an HMD may include an outwardly-directed speaker that allows interaction between a digital assistant integrated with the HMD and an outside observer. These types of output systems and techniques, when integrated with an HMD (either alone or in any combination), allow more varied and robust interactions between HMD wearers and other people.

The HMD may also have outward-facing cameras or other sensors allowing the wearer of the HMD to see video or images of the real world, including people, via the internal displays of the HMD. (Such video or images may be displayed to the wearer without any modification, or they may be fully or partially integrated into the wearer's virtual environment.) Because each person can see the face and eyes of the other, the dual-display HMD effectively provides a two-way pathway for visual communication between the wearer and a person in the real world environment. Moreover, both people can experience more natural communication without requiring the wearer of the HMD to completely disengage with the virtual experience.

Apple's patent FIGS. 3A-C below illustrates the transition to Apple's full 'EyeSight' feature.  

In some cases, the outward-facing display #106 displays visual output that is not suggestive of eyes or other facial features, yet still conveys information to observers and increases the quality and the content of interactions between a wearer and an observer. Such visual outputs may include, for example, symbols or images indicative or suggestive of emotions or moods, informational displays (e.g., current weather, a description of the content the wearer is viewing, the wearer's calendar), or patterns that indicate operational states or a status of the wearer or the wearer's virtual environment. These are presented below in FIGS. 9A-D.

Apple's patent FIGS. 10A-10D below show examples of symbolic visual outputs that convey information about the wearer 100 and/or the wearer's state of interaction with virtual and real-world environments. The visual outputs shown in FIGS. 10A-10D may be part of a communication convention for outward-facing displays to convey a set of concepts that may help improve interactions between the wearer 100 and external observers. 

Apple's patent FIGS. 10A-10D above show examples of symbolic visual outputs that convey information about the wearer and/or the wearer's state of interaction with virtual and real-world environments. FIG. 10A shows the HMD displaying a visual output #1000 of a triangle, which may resemble a “playback” symbol. This visual output may indicate that the wearer is fully engaged in a virtual environment and/or that the real-world environment is not being presented to the wearer.

In patent FIG. 10B we see the HMD displaying a visual output #1002 with two lines or shapes that are suggestive of closed eyes. The visual output may convey that the HMD is in a “do not disturb” state or that the wearer otherwise does not wish to be interrupted or engage with the real-world environment.

FIG. 10C shows the HMD displaying circles that are suggestive of open eyes (similar to the visual output 804, described above), indicating that the wearer is viewing the real-world environment and/or is open to engage with individuals in the real-world environment.

FIG. 10D shows the HMD displaying a visual output #1006 of circles with inset icons or symbols representing video cameras, indicating to outside observers that the HMD is recording images (e.g., video or still images) of the real-world environment. Other symbols or colors may instead be used to convey that the HMD 102 is recording, such as red circles or red circles with a contrasting peripheral border.

FIG. 11A above shows the HMD with light arrays #1100 that each include nine light sources arranged in a pattern. (Amounts and patterns of light sources other than those shown in FIG. 11A may also be used.) The light sources in the light arrays #1100 may be LEDs, incandescent light sources, or any other suitable light source.

FIG. 11B shows the light arrays displaying a visual output #1102 suggestive of open eyes. This pattern may be displayed when the wearer is viewing or is able to view the real-world environment and/or is willing to engage with others.

FIG. 11C shows the light arrays displaying a visual output #1104 suggestive of closed eyes, which may be may displayed when the HMD is in a “do not disturb” state or the wearer 100 otherwise does not wish to be interrupted or engage with the real-world environment.

FIG. 11D shows the light arrays displaying a visual output #1106 having the shape of two “X” symbols, which may indicate that the wearer is fully engaged in a virtual environment and/or that the real-world environment is not being presented.

FIGS. 12A-12F below show additional examples of how a level of engagement of a wearer with the real-world environment may be conveyed using different types of visual outputs on an outward-facing display. For example, the particular visual output that is displayed on an outward-facing display may be indicative or suggestive of, for example, what the wearer is viewing, the level of virtualization and/or immersion of the wearer's environment, or the like.

With reference to FIGS. 12A and 12B, when a wearer is fully engaged in a virtual environment #1202, such as a video game (as shown in FIG. 12A) or a movie or video, the outward-facing display may show a visual output 1204 including logo or title (as shown in FIG. 12B) that conveys information about what the wearer is engaged with, as well as conveying the fact that the wearer is fully engaged with the virtual environment (and thus may not be aware of the real-world environment).

FIG. 12B shows a logo representative of the video game, other symbolic outputs representative of the content being displayed to the wearer may be displayed, including text (such as a movie title), stylized text, an image of a character in a movie, a movie poster or other promotional images, etc.

Apple's patent FIGS. 12C-D illustrates Apple's full EyeSight feature; FIG. 12E shows an example mixed reality visual output 1212 (displayed to the wearer) in which a real-world element #1210 (e.g., a person) is incorporated into a virtual environment. In order to convey to observers that the wearer 100 can see them or some other aspect of the real-world environment, the HMD may display, on the outward-facing display, the wearer's eyes embellished with an additional symbol or indicator that the wearer's view is somehow modified or mediated. For example, as shown in FIG. 12F, a visual output #1214 may include real-time images of part of the wearer's face with circles around the eyes. Increasing the amount of information that observers have about what the wearer is viewing may help to ease awkwardness and generally increase the comfort level of outside observers, as observers may feel more comfortable knowing when and whether the wearer is being exposed to content in addition to the real-world environment. 

Apple's patent 14A-D illustrates Vision Pro's use of a lenticular display that allows the external viewer to see the headset user's face move naturally depending on the angle viewed.

For more details, review Apple's granted patent 11861255.