Today the U.S. Patent and Trademark Office officially granted Apple a patent that relates to head-mountable devices, and, more particularly, to outputs provided by a head-mountable device to help a user distinguish between virtual and real objects. To control the objects, the patent also covers a series of external handheld or worn controlling devices to work with the HMD.

A head-mountable device can be worn by a user to display visual information within the field of view of the user. The head-mountable device can be used as a virtual reality (VR) system, an augmented reality (AR) system, and/or a mixed reality (MR) system. A user may observe outputs provided by the head-mountable device, such as visual information provided on a display. The display can optionally allow a user to observe an environment outside of the head-mountable device. Other outputs provided by the head-mountable device can include audio output and/or haptic feedback. A user may further interact with the head-mountable device by providing inputs for processing by one or more components of the head-mountable device. For example, the user can provide tactile inputs, voice commands, and other inputs while the device is mounted to the user's head.

Head-mountable devices, such as head-mountable displays, headsets, visors, smartglasses, head-up display, etc., can perform a range of functions that are managed by the components (e.g., sensors, circuitry, and other hardware) included with the wearable device.

While a head-mountable device can provide outputs to a user in a variety of ways, it can also be helpful to allow a head-mountable device to receive inputs from a user. A head-mountable device can provide a user experience that is immersive while allowing the user to interact with the system in a manner that is natural, familiar, and intuitive.

A head-mountable device can be operated to provide a view of physical objects in a physical environment as well as virtual objects that are generated by the head-mountable device.

The virtual objects may not have a corresponding physical object in the physical environment. The head-mountable device can provide an immersive experience in which many or all of the objects (e.g., both physical and virtual) are perceived as being real.

However, under certain conditions, it may be desirable for a user to distinguish between the physical objects and the virtual objects. Indications provided by the head-mountable device can help a user make choices that enhance the user's comfort and control by improving the user's awareness of physical objects and/or virtual objects. The indications can further help a user know how to successfully interact with one or more physical objects and/or virtual objects.

It may be useful to provide indications that help the user distinguish between the physical objects and the virtual objects when desired. The indications can be visually applied to physical objects and/or virtual objects within a view provided by the head-mountable device.

The indications can be provided to notify the user of objects outside the view provided by the head-mountable device. The indications can be visual, auditory, and/or haptic in nature.

The indications can be provided based on a user's detected action, intention, and/or attention so that the indication is more targeted to a user's needs. The indications can be provided under particular conditions, so that the user can more easily maintain an immersive experience when the indications are not desired.

Apple’s patent FIG. 1 below illustrates a perspective view of a head-mountable device (HMD) and external hand-held devices for operation by a user. The external holdable or wearable devices #200 can be devices for operation by a user. These external devices can be input and/or output devices such as a game controller, a remote control, a trackpad, a mouse, a trackball, finger devices and the like.

Additionally, or alternatively, the external devices can be electronic devices, such as a phone, a tablet computing device, a mobile computing device, a watch, a laptop computing device, a stylus, a digital media player, wearable device (clothing, gloves, shoes, jewelry, apparel, and the like), a display, a television, and the like.

Apple’s patent FIG. 2 illustrates a block diagram of a head-mountable device and an external device

As shown in FIG. 3 above, a user (#10) can wear the head-mountable device #100, which provides a field-of-view 90 of a physical environment. A physical object #20 can be located within the field-of-view 90.

The head-mountable device can perform object recognition with respect to the physical object #20. For example, the view can be captured and analyzed by the head-mountable device to determine a characteristic of the physical object.

An eye-tracking sensor of the head-mountable device can be used to determine which of multiple objects within a field of view of the head-mountable device corresponds to an object that is the focus of the user's gaze.

Apple’s patent FIG. 5 above illustrates a display of a head-mountable device providing a view of a physical object and a virtual object with a visual indicator #40.

Apple’s patent FIG. 10 below illustrates a top view of a head-mountable device in use by a user and a trajectory of a virtual object as displayed by the head-mountable device; FIG. 11 illustrates a side view of a head-mountable device in use by a user and a trajectory of a virtual object as displayed by the head-mountable device. 

Apple’s patent FIG. 14 above illustrates a top view of a head-mountable device in use by a user and an output provided by external handheld or wearable devices.

For more details, review Apple's granted patent 11,361,735.