Today the US Patent & Trademark Office published a patent application from Apple that relates to a future head-mounted display device hailing/alert system. When a user is playing a VR game and is deeply engaged with visuals and audio, Apple has devised a system whereby someone entering the room may call out a command as simple as "Excuse me" and the gamer will instantly be given an on-display alert that someone, or someone specific via facial recognition, needs your attention. The alternative is that the game is stopped and the display goes transparent so that the gamer could see who needs their attention. It's a superior way to gently break a gamer's concentration instead of shocking them with a tap on the head or shoulder.

Apple initially states that a head-mounted display that may be used to present a computer-generated reality application (e.g., including audio and video signals) to a user wearing the head-mounted display.

While using a computer-generated reality application, a user's ability to sense and be aware of their physical surroundings can be substantially impaired. Using a computer-generated reality application could make it difficult for other people near the user to get the attention of the user.

Apple's solution includes delivering systems and processes for facilitating the hailing of user immersed in computer-generated reality application.

The systems may include image sensors and/or microphones attached to the head-mounted display or positioned nearby so as to capture images and/or sound from a vicinity around the user wearing the head-mounted display.

For example, data from these sensors may be analyzed to detect a person and/or indications that the person is trying to hail the user, e.g., to attract the attention of the user.

For example, if the person speaks to the user, audio triggers (e.g., key phrases such as "hello," "excuse me," or a name of the user) can be recognized in the speech signal.

In some implementations, a direction of arrival for a speech signal may be determined and compared to the view angle for the person to verify that the speech signal is coming from the nearby person.

In some implementations, a face of the person (e.g., including one or more facial landmarks, such as a mouth, a nose, or pupils) may be detected recognized as matching a registered face of a known person (e.g., a friend or colleague of the user).

In some implementations, the face can be detected and an orientation of the face with respect to the head-mounted display can be determined to assess whether the person is facing toward the user. In some implementations, eyes of the person can be analyzed more closely to determine a gaze direction, in order to assess whether the person is looking at the user.

Based on one or more of the indications of hailing described above in the sensor data, a hail event can be detected. When a hail event is detected, an alert may be presented to the user using the head-mounted display. For example, an audio alert message may be presented using headphones of the head-mounted display.

For example, an alert image (e.g., text and/or an image of the person detected) may be presented using a head-mounted display. In some implementations, the audio and/or video of the computer-generated reality application may be transitioned to a transparent to enhance the ability of the user to sense and be aware the surrounding physical environment.

Using the described systems and processes for computer-generated reality hailing may provide advantages over some conventional systems for providing computer-generated reality experiences to users. For example, social experiences (e.g., gaming experiences) may be improved by allowing more natural interactions with people near the user during use of a computer-generated reality application.

Apple's patent FIG. 1 below is an illustration of an example of a hailing scenario; FIG. 4 is a block diagram of an example of a system configured to enable hailing of a user wearing a head-mounted display.

Apple's patent FIG. 5 above is a flowchart of an example of a process for facilitating hailing of a user wearing a head-mounted display.

Apple's patent application '0392830 that was published today by the U.S. Patent Office was filed back in Q2 2019 with some work being done in 2018. The depth of Apple's patent base for future smartglasses and Head-Mounted Displays can be viewed in our Special Archives.

The sole inventor is listed as being Golnaz Abdollahian: AI & ML Engineering Manager.

Considering that this is a patent application, the timing of such a product to market is unknown at this time.