Apple won a series of Eyewear Patents today covering the use of ultrasonic sensors, being able to hail the attention of a user wearing an HMD+
The U.S. Patent and Trademark Office officially published a series of 52 newly granted patents for Apple Inc. today. In this particular report we briefly cover a series of patents relating to Apple's future eyewear devices. The most outstanding granted patent in this group was covered in a separate report this morning covering Eye Tracking ID for eyewear. The remaining three eyewear patents in this report cover the use of an ultrasonic sensor, a display device with mechanically adjustable optical combiner and the ability to hail a person wearing an MR headset.
Ultrasonic Sensors for an MR Headset
In this first granted patent Apple notes that electronic devices provide users with many ways to interact with the world around them. For example, users may interact with electronic devices using electronic devices which may aid in detecting and identifying objects in an environment. For example, some devices provide visual-based object detection, in which cameras are utilized to detect and identify an object in an environment.
However, some items are difficult to detect in an environment. For example, items may be outside a field of view of a camera, or may be difficult to detect by a camera. As an example, transparent or glass walls, doors, and the like may be difficult to detect by visual-based object detection. Mirrors may be another example of a type of item that may be difficult to detect using visual-based object detection. Accordingly, what is needed is an improved system and method to detect objects in an environment.
Apple's granted patent covers a method future eyewear to detect objects more accurately. The method may include transmitting, by a first device, a first pulse into an environment using a wide field configuration of an ultrasonic sensor array, detecting a presence of an object in the environment based on a detected echo based on the first pulse.
In response to detecting the presence of the object, a targeted configuration of the ultrasonic sensor array is determined, and a second pulse is transmitted into the environment based on the second pulse, wherein a characteristic of the object is determined based on a detected echo from the second pulse.
Another way that ultrasonic sensors may be used on a head mounted device are to enhance a user experience. For example, glass tabletops and other surfaces may make accurate placement of virtual objects in an environment challenging. By utilizing ultrasonic sensors, real world items in the environment may be better identified and located such that virtual objects may be more accurately placed in the environment.
Apple's granted patent FIG. 2A below is an example electronic device #200 comprising a set of ultrasonic sensors. As shown, the device may include a uniform or non-uniform array of sensors. For purposes of the example shown, ultrasonic sensors with the capability of transmitting and receiving signals may be distributed across three sides of an electronic device; FIG. 2B shows an example chart 250 illustrating the field of view of various combinations of the ultrasonic sensors shown above may be utilized in various combinations to manipulate a field of view.
For more on Apple's granted patent 11,002,851 titled 'Ultrasonic Sensor' click here.
Display Device with Mechanically Adjustable Optical Combiner
In this second granted patent Apple describes optical systems that may be used to present images to a user. In some mixed reality systems, displays present computer-generated content that is overlaid on top of real-world images. An optical system may use an optical combiner to combine real-world image light with image light from a display.
The optical combiner may be a curved partially reflective mirror that is mounted in front of a user's eyes using head-mounted support structures. The head-mounted support structures may be configured to form a head-mounted device that is worn on the head of a user. In some arrangements, the head-mounted device may include a non-removable display. In other arrangements, the head-mounted device may be configured to receive a removable portable electronic device with a display such as a removable cellular telephone.
During operation, one or more actuators may be used to mechanically adjust the optical combiner. The optical combiner may, for example, be tilted about a hinge axis, rotated around a pivot point, flexed to provide the optical combiner with a desired amount of curvature, and/or may otherwise be mechanically adjusted in shape and/or location. These mechanical adjustments may adjust the focal length of a reflective lens formed from the optical combiner, may steer reflected image light, and/or may otherwise be used in enhancing the ability of users to view content while using a head-mounted device that contains the optical combiner. For example, actuator movements of the optical combiner (tilting, rotating, bending, bowing, etc.) may be used in adjusting the lens power associated with the mirror lens that is formed from the partially reflective coating on the optical combiner and thereby adjusting virtual image distances associated with computer-generated objects being overlaid on the real world.
Apple's patent FIG. 7 below is a cross-sectional side view of an illustrative optical combiner and an associated rotating system coupled to a hinge that is used to rotate the optical combiner.
For more on Apple's granted patent 11,002,971 click here.
Alerts during an Immersive MR Experience
In this third granted patent of the day relating to future eyewear, Apple notes that while using a computer-generated reality application, a user's ability to sense and be aware of their physical surroundings can be substantially impaired. Using a computer-generated reality application could make it difficult for other people near the user to get the attention of the user. Systems and processes for facilitating the hailing of user immersed in computer-generated reality application are described in today's granted patent.
When a hail event is detected, an alert may be presented to the user using the head-mounted display. For example, an audio alert message may be presented using headphones of the head-mounted display. For example, an alert image (e.g., text and/or an image of the person detected) may be presented using a head-mounted display.
In some implementations, the audio and/or video of the computer-generated reality application may be transitioned to a transparent to enhance the ability of the user to sense and be aware the surrounding physical environment.
Apple further describes systems that may include image sensors and/or microphones attached to the head-mounted display or positioned nearby so as to capture images and/or sound from a vicinity around the user wearing the head-mounted display.
For example, data from these sensors may be analyzed to detect a person and/or indications that the person is trying to hail the user, e.g., to attract the attention of the user.
For example, if the person speaks to the user, audio triggers (e.g., key phrases such as "hello," "excuse me," or a name of the user) can be recognized in the speech signal.
In some implementations, a direction of arrival for a speech signal may be determined and compared to the view angle for the person to verify that the speech signal is coming from the nearby person.
In some implementations, a face of the person (e.g., including one or more facial landmarks, such as a mouth, a nose, or pupils) may be detected recognized as matching a registered face of a known person (e.g., a friend or colleague of the user).
In some implementations, the face can be detected and an orientation of the face with respect to the head-mounted display can be determined to assess whether the person is facing toward the user.
In some implementations, eyes of the person can be analyzed more closely to determine a gaze direction, in order to assess whether the person is looking at the user.
Apple's patent FIG. 1 below is an illustration of an example of a hailing scenario; FIG. 4 is a block diagram of an example of a system configured to enable hailing of a user wearing a head-mounted display.
For more on Apple's granted patent 11,002,965 click here.