Today the US Patent & Trademark Office published a patent application from Apple that relates to a future mixed reality headset device and more specifically to object tracking for an HMD. A future HMD or Glasses will use sensors that could recognize and track objects, surfaces and in-air gesture commands.

Apple's invention covers Head-mounted devices, such as head-mounted displays, headsets, visors, smartglasses, head-up display, etc., can perform a range of functions that are managed by the components (e.g., sensors, circuitry, and other hardware) included with the wearable device.

Head-mounted devices can be equipped with a wide range of outward and inward facing sensors. These sensors can recognize and track objects, surfaces, and user gestures such as hand and body movements.

Head-mounted devices could collect data from and/or relating to a device and make certain determinations that aid the process of displaying a representation (e.g., virtual rendering) to the user.

For example, an object can be provided with indicators that allow a head-mounted device to determine both an identity and a characteristic (e.g., position, orientation, distance, etc.) of the object. Accordingly, the same indicators can be used to determine the information necessary to produce a virtual representation of the object in a manner that resembles the physical object.

By further example, an electronic device that is recognizable by a head-mounted device could be attached to another object in a manner that maintains a fixed relative position and orientation between the electronic device and the object.

The head-mounted device could determine both an identity and a characteristic (e.g., position, orientation, distance, etc.) of the electronic device. The information regarding the electronic device can be used to produce a virtual representation of the object in a manner that resembles the physical object.

By further example, an electronic device could be attached to another object and collect data that is transmitted to the head-mounted device for analysis.

The head-mounted device could determine both an identity and a characteristic (e.g., position, orientation, distance, etc.) of the electronic device based on the data. The information received from the electronic device can be used to produce a virtual representation of the object in a manner that resembles the physical object.

Apple's patent FIG. 1 illustrates a perspective view of a head-mounted device; FIG. 2 illustrates a view of a head-mounted device, a personal device, and an electronic device; FIG. 4 illustrates a display of a head-mounted device providing a representation of a personal device.

Apple's patent FIG. 7 illustrates a perspective view of an electronic device and a personal device; FIG. 8 illustrates a perspective view of an electronic device attached to a personal device; FIG. 9 illustrates a perspective view of an electronic device attached to a personal device; FIG. 12 illustrates a perspective view of an electronic device attached to a personal device.

Apple's patent FIG. 11 above is an electronic device that can be attached to an object to facilitate determination of a characteristic thereof by a head-mounted device.

The electronic device #900 above can include any electronic device that is in communication with a head-mounted device. For example, the electronic device 900 can be an input device for operation by a user. While the electronic device 900 is shown as a stylus, it will be understood that other types of electronic devices are contemplated, such as a phone, a tablet computing device, a mobile computing device, a laptop computing device, a game controller, a watch, a digital media player, a wearable device (clothing, gloves, shoes, jewelry, apparel, and the like), a display, a television, and the like.

It will be understood that the electronic device #900 can serve purposes and provide functions other than detectability by a head-mounted device. As such, the electronic device #900 can be a device that provides a set of functions when used with the object #990 and another set of functions when not used with the object #990.

The object #990 can be any object that is capable of being represented (e.g., visually) by the head-mounted device. For example, the object 990 can be any object in an external environment for representation by the head-mounted device in a virtual environment.

Additionally, or alternatively, the object #990 can be any object that can be operated and/or manipulated by a user. While the object #990 is shown as a sport equipment items (e.g., tennis racquet), it will be understood that other types of objects are contemplated, such as a tool, an instrument, a game controller, and the like.

Such items can be held and/or manipulated by a user during use with the HMD. For example, the position, orientation, movement, and/or other manipulation of the object #990 in space can be interpreted as a user input.

Additionally, or alternatively, the object can be an input device. For example, the object can be a keyboard, a trackpad, a mouse, a trackball, a game controller, a remote control, a stylus, a joystick, and the like. Such items can be held and/or manipulated by a user during use. For example, the user can operate an input component of the object #990 in a manner that is interpreted as an input.

As shown in FIG. 11, the object #990 can include an object attachment element #992. The electronic device #900 can be releasably attached to the object #990 via the object attachment element #992 (and/or a feature of the electronic device, such as an electronic device attachment element). While attached to each other, the electronic device 900 and the object 990 can maintain fixed relative positions and orientations.

Review Apple's patent application number 20210019919 for finer details. Considering that this is a patent application, the timing of such a product to market is unknown at this time.