Today the U.S. Patent and Trademark Office officially granted Apple a patent relating to an AR/VR headset working in sync with a secondary device for a game for instance, wherein the second device could be seen in a virtual world as a sword, gun or other apparatus.

Apple notes that Head-mounted devices ("HMD") (also known as headsets) are often used in augmented reality ("AR") and virtual reality ("VR") systems to present a user with virtual graphical objects that either complement or replace a surrounding real-world environment in the user's field of view.

To enhance that virtualized experience, AR/VR systems may include a secondary device that enables the user to interact with the virtual graphical objects. For example, the secondary device may correspond to a sword or other implement in the context of the virtualized experience.

The AR/VR system may update a depiction of the sword in the virtualized experience to mirror a movement of the secondary device by tracking a correspondence between the secondary device and the HMD or another device.

Apple's granted patent covers devices, systems, and methods for determining correspondences between a head-mounted device ("HMD") and a secondary device physically unassociated with the HMD using event camera data. One example method involves obtaining, at an HMD with an event camera, light intensity data indicative of an intensity of light incident on a plurality of pixel sensors from a stream of pixel events output by the event camera.

The plurality of pixel sensors are disposed on the event camera at known locations relative to an HMD reference frame. At least a portion of the light is emitted by a plurality of optical sources disposed on the secondary device at known locations relative to a secondary device reference frame.

Each respective pixel event is generated in response to a particular pixel sensor detecting a change in light intensity that exceeds a comparator threshold. A set of optical sources among the plurality of optical sources that are visible to the event camera are identified by recognizing defined illumination parameters associated with the set of optical sources using the light intensity data. Location data is generated for the set of optical sources in the HMD reference frame using the light intensity data.

The correspondence between the secondary device and the HMD is determined by mapping the location data in the HMD reference frame to respective known locations of optical sources of the set of optical sources relative to the secondary device reference frame.

Apple's patent FIG. 1 represents an example operating environment #100 represents the various devices involved in presenting a user with an augmented reality ("AR") experience and/or a virtual reality ("VR") experience.

As depicted in the example of FIG. 1, operating environment 100 includes a controller #110, a head-mounted device ("HMD") #120, and a secondary device #130.

To present a user with an AR experience, natural sensory information that the user perceives from a real-world environment proximate to the user (e.g., scene #105) is overlaid with virtual sensory information (e.g., two-dimensional ("2D") and/or three-dimensional ("3D") graphical content, sounds, tactile feedback, and the like) generated or at least processed by a computing device.

That is, the virtual sensory information that is presented to a user as part of an AR experience ("augmented content") coexists with natural sensory information of the real-world environment. For example, the real-world environment, as represented by scene #105, may include object #140 which represents a chair and the augmented content is a video game character in this example, the user may see the video game character sitting in the chair as part of an AR experience.

Apple's patent FIG. 9 above is a block diagram of an example of the HMD (#120) in accordance with some implementations.

Apple's patent FIG. 7 below is a flow-chart illustrating an example of a method for determining a correspondence between an HMD and a secondary device physically unassociated with the HMD.

For more details review Apple's granted patent 10,845,601 that was published today by the US Patent and Trademark Office.