Apple Patent dives deep into Future Mixed Reality gaming with Sophisticated Game Controllers & Next-Gen Console
Today the US Patent & Trademark Office published a pair of head mounted device patent applications from Apple. The first patent application is a major one focused on AR, VR, and MR Gaming. The filing is focused on Optical Tracking and a Panoramic Optical Scanner. Apple touches on a possible future gaming console that is quite advanced. Apple has been adding game console to a number of trademark applications in the last year. Apple's patent discusses gaming controllers in-depth and yet doesn't provide an image of the main controllers that Apple claims are held in each hand. They do however, illustrate an alternative future Apple Watch that could be equipped with a optical tracking system so that wrist gestures could be used like a game controller.
Apple notes in their patent background that there are scenarios in which it is desirable to determine ranging information (i.e., relative locations and distance between) and/or relative posing information (i.e., relative locations and orientations) for a pair of devices.
For example, when playing an augmented reality (AR), virtual reality (VR), or mixed reality (MR) game, it may be desirable to determine a range or posing of a game controller with respect to a game console.
In some cases, it may be desirable to determine ranges or posings of multiple game controllers, goggles, and so on with respect to the game console, and to determine ranges or posings of the game controllers with respect to each other.
As another example, it may be desirable to determine ranging and/or relative posing information for devices that engage in line-of-sight optical communication (e.g., in a room or around a conference table).
Optical Tracking & Panoramic Optical Scanner
Apple's invention covers embodiments of the systems, devices, methods, and apparatus directed to optical tracking. In accordance with described techniques, a panoramic optical scanner may be constructed with fewer and/or lighter weight moving parts.
A panoramic optical scanner may also be constructed with components disposed along and about a single axis, with the components being disposed to receive, direct, or steer electromagnetic radiation (e.g., visible or invisible (e.g., infrared (IR)) light) along, about, or outward from the single axis.
Panoramic optical scanners electromagnetic radiation having a set of electromagnetic radiation edges into a panoramic field of view. For example, a panoramic optical scanner may direct ring-shaped electromagnetic radiation to one side of an axis, and sweep the ring-shaped electromagnetic radiation about the axis.
An optical tracker may be provided with a set of multiple photodetectors capable of detecting the electromagnetic radiation edges emitted by a panoramic optical scanner, and may identify timings of the edges and relate the timings to a timing reference of the panoramic optical scanner.
AR, VR, Mixed Reality Gaming
One environment in which a panoramic optical scanner and one or more optical trackers may be used is an AR, VR, or MR game, in which each participant may wear a pair of goggles and hold a number of game controllers (e.g., one or more of game controllers).
By way of example, each game controller may have a circular sensing component coupled to a handle, and each participant may hold a game controller in each hand.
Unfortunately Apple doesn't provide an image of their game controllers to be held in each hand. Of course the concept is common and found in systems like the Oculus Rift as shown below.
Alternatively, each participant may hold or otherwise be associated with no, one, or multiple game controllers, and different participants may be associated with different numbers of game controllers. Alternatively, a participant may not wear a set of goggles and instead just hold or manipulate one or more game controllers which could be smartphones.
In some embodiments, a game controller may have an alternate configuration or shape. For example, a game controller may be shaped like a particular item it represents, such as a weapon, a glove, or a steering wheel. Also, some game controllers may be held by two or more participants at once, or may be items that are worn by, sat on, stood on, or ridden by a participant.
An advanced Gaming Console
Apple notes that the game played by the participants may be hosted by a game console or other device, within or remote from the environment in which the game is played. By way of example, a game console may sit on a table within a room in which the participants are playing.
In some embodiments, the game console may wirelessly serve game video to goggles worn by the participants, or the game console may serve game video over a wireless network to which the game console or goggles are connected.
Audio for the game may be served in the same manner, or provided via a speaker of the game console, or broadcast from wired or wireless speakers connected to the game console or a wireless network. In some embodiments, the game console may be a device that performs other functions.
For example, the game console may also function as one or more of a stereo receiver, a computer, a personal assistant, a security system, and so on.
While the game is played, the game may need to determine posings (positions and orientations) of the goggles, the game controllers, and/or the game console with respect to each other (i.e., relative posings). The posings may enable these items (or virtual representations thereof) to be posed or manipulated within an AR, VR, or MR game environment, or may enable scenes, structures, things, persons, animals, and so on to be rendered in relation to the posings of the goggles, game controllers, and/or game console.
In some embodiments, the relative posings of the goggles, game controllers, and/or game console, with respect to each other, may be determined by operation of a panoramic optical scanner included in the game console and optical trackers included in each of the goggles and game controllers. In other embodiments, the posings of the goggles, game controllers, and/or game console may be determined by operation of a panoramic optical scanner included in one or more of the goggles or game controllers, and optical trackers in the remaining devices.
Typically, the effects of including a panoramic optical scanner or optical tracker in a mobile device can be ignored when determining relative posings of the devices, given that the speed of optical communication between mobile devices (or a mobile device and a stationary device) at close range is many orders of magnitude greater than the speeds at which mobile devices may be humanly moved.
In some embodiments, a panoramic optical scanner may have a panoramic field of view of 360 degrees, 270 degrees, or some other angular extent.
For example, a panoramic optical scanner incorporated into the top of the game console may have a 360 degree panoramic field of view, whereas a set of goggles may incorporate a panoramic optical scanner into each temple portion of the goggles, with each panoramic optical scanner having a 270 degree panoramic field of view that overlaps the 270 degree panoramic field of the other panoramic optical scanner.
Once again, Apple keeps this aspect of the patent a secret in respect to what the console could look like, even conceptually.
Apple Watch as a Possible Game Controller
Apple's patent FIG. 9 below shows us an example controller 900 (e.g., a game controller, a gesture input device, etc.) in the form of an Apple Watch that may incorporate an optical tracker including photodetectors distributed, in sets or individually, at various locations #902 (e.g., locations 902a, 902b, 902c, 902d, and 902e) over various surfaces of the controller 900. By way of example, the controller #900 is shown to be a wearable device, such as a watch, fitness monitor, or health monitor. Alternatively, the controller may be configured, for example, as a bracelet, sleeve, or handheld device.
Orienting the photodetectors with their surfaces perpendicular to three or more axes (and preferably, but not necessarily, orthogonal axes), can enable movement of the controller or gestures made by its wearer, to be tracked with six degrees of freedom.
HMD / Goggles
A set of goggles may also incorporate an optical tracker including photodetectors. In some embodiments, the controller or goggles may include sets of four or more photodetectors at various locations 902a, 902b, 902c, 902d, 902e.
Photodetectors may be distributed over more or fewer surfaces of the controller or goggles, or in a higher or lower density over one or more surfaces of the controller or goggles. Alternatively, the controller 900 or goggles may include a panoramic optical scanner at one or more of the locations 902, or at other locations.
For example, a set of goggles may include a panoramic optical scanner at each of its temples. When a device such as a set of goggles has multiple panoramic optical scanners, the panoramic optical scanners may have overlapping panoramic fields of view (e.g., overlapping 270 degree fields of view).
When the controller #900 is a wearable device having a body #904 (e.g., a watch body) and a band #906 (e.g., a watch band), photodetectors may be disposed in the body and/or in the band.
Apple's patent FIG. 15C below is one example of a panoramic optical scanner #1500; FIG. 16C show various stages of a process that may be performed by a panoramic optical scanner or an optical tracker.
Apple's patent application 20200057147 that was published today by the U.S. Patent Office was filed back in Q3 2019. Considering that this is a patent application, the timing of such a product to market is unknown at this time.
Mark Winkler: Engineering Manager, Optical Sensor Design
Tong Chen: Optical Sensing Engineer
Wenrui Cai: Engineering Manager
A second Apple HMD patent application was published today the U.S. Patent Office titled "Display Illumination Systems" under #20200057308.
Apple notes that "A head-mounted device such as a pair of augmented reality glasses that is worn on the head of a user may be used to provide a user with computer-generated content that is overlaid on top of real-world content. The real-world content may be viewed directly by a user through a transparent portion of an optical system. The optical system may be used to route images from one or more pixel arrays in a display system to the eyes of a viewer.
A waveguide such as a thin planar waveguide formed from a sheet of transparent material such as glass or plastic or other light guide may be included in the optical system to convey image light from the pixel arrays to the viewer. This type of waveguide may also be used to carry light in an illumination system that is configured to provide illumination to a display system. The display system may include reflective displays such as liquid-crystal-on-silicon displays, microelectromechanical systems (MEMs) displays, or other displays."
For more information on this patent application, review patent application 20200057308 here.