Today the US Patent & Trademark Office published a patent application from Apple that relates to interacting with and manipulating a user interface of a Mixed Reality Headset, iPad or Mac display using a future Apple Pencil that could be used as a VR gaming controller to control a virtual agent (gaming character) or managing interactions with physical objects. The patent also covers controlling the movement of objects using gaze controls.

In Apple's patent FIGS. 1 and 6B below we see a user holding an Apple Pencil, which in this context, is used as a gaming controller to move objects or a gaming character which Apple refers to as a "Virtual Agent" #606. The functions of the Apple Pencil will include inputs involving tapping, rolling, twirling, flicking and swiping on a touch surface of the Pencil.

The Apple Pencil/control device #130 includes a sensor or set of sensors that detect inputs from the user based on haptic and/or tactile contact with the touch-sensitive surface #175. In some implementations, the Apple Pencil/control device includes any of a plurality of touch sensing technologies including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive surface.

In the other half of the patent, Apple goes into detail about using eye/gaze tracking as a means of moving objects in an extended reality (XR) environment or game.

Apple's patent FIG. 5A is a block diagram of a first portion of an example content delivery architecture while FIG. 5C is a block diagram of a second portion of the example content delivery architecture; FIG. 5B illustrates example data structures.

This an in-depth patent that mixed reality developers will appreciate. To review its greater details, review Apple's patent titled "Managing Interactions Directed To A User Interface With A Physical Object" under application number US 20230042447 A1.

Apple Inventors

• William Lindmeier: Manager, Interaction Designer
• Ben Jackson: Engineering Manager
• Brent Baugh: Hardware Design and Prototyping
• Matt Sundstrom: On Apple Design Team: Human Interface
• Israel Pastrana Vicente: Human Interface Designer

Mixed Reality Developers may also want to checkout another patent published today titled "Contextual Computer-Generated Realty (CGR) Digital Assistants" under patent application US 20230045634 A1.