Today the U.S. Patent and Trademark Office officially granted Apple a patent that relates to allowing an iPhone to be an input device for a mixed reality headset. More specifically, allowing the user to see their iPhone display and interact with it within computer-generated reality (CGR) environments like a game. The iPhone could then be used as a three-dimensional (3D) controller, 3D pointer, user interface input device, etc.).

In their patent background, Apple notes that to enable user interactions with electronic content, it may be desirable to enable a user to provide input via a separate real-world device, such as an iPhone's (or iPad's) the touch screen.  However, existing systems are unable to adequately track the locations of a smartphone or tablet relative to the content in the CGR environment. This is what Apple's granted patent solves.

Apple's patent covers devices, systems, and methods that provide improved user interfaces for interacting with electronic content using multiple electronic devices. Some implementations involve a first device (e.g., a head-mounted device (HMD)) that has an image sensor (e.g., a camera) and one or more displays, as well as a second device (e.g., an iPhone) that has a display.

A marker is displayed on the display of the second device (an iPhone) and the first device determines a relative position and orientation of the second device to the first device based on the marker.

In some implementations, a marker is an image containing texture/information that allows the image to be detected and makes it possible to determine the image's pose with respect to a camera.

In some implementations, a control signal is generated based on the relative position and orientation of the second device (e.g., the first device uses the position and orientation of the second device to enable the second device (an iPhone) to be used as a three-dimensional (3D) controller, 3D pointer, user interface input device, etc.).

Apple's patent FIG. 2 below illustrates a user being able to use an iPhone as an input device in computer-generated reality (CGR) environments while wearing a Mixed Reality Headset.

Apple's patent FIG. 3A above illustrates a marker #310 displayed by the physical second device #130 (an iPhone) as noted in FIG. 2. In some implementations, the user is unable to view the physical display of their iPhone because the user is immersed in the virtual scene #205. Accordingly, in some implementations, the iPhone displays a marker (#310) on the physical display of the iPhone to facilitate tracking of the iPhone by the Head Mounted Display (HMD).

In some implementations, the marker serves as a reference point for the HMD to accurately track the location and rotation of the iPhone.

In some implementations, the marker (#310) is an image containing texture/information that allows the image to be detected and makes it possible to determine the image's pose with respect to a camera.

In some implementations, the marker is a pattern and, in some implementations, the marker is a singular indicator. For example, the marker may include a grid, cross-hatching, quadrant identifiers, screen borders, etc.

For more details, review Apple's granted patent 11,036,284