The U.S. Patent and Trademark Office officially published a series of 64 newly granted patents for Apple Inc. today. In this particular report we cover Apple's patent that reveals an Apple Pencil accessory can be used to receive inputs and track motion of a user in three-dimensional space such as AV/AR environments when using a future Head Mounted Display Device HMD.

The stylus/Apple Pencil #100 can be used in a real environment to influence a mixed reality system. For example, a device providing a visual display or other output to a user can apply inputs from the stylus to a mixed reality system.

The inputs can be applied to augmented reality system, an augmented virtuality system, and/or a virtual environment.

By further example, the stylus can be used to virtually mark and/or annotate items from a real environment and/or from a virtual environment. The markings and/or annotations can be recorded within the mixed reality system and output to a user.

The tip sensor can detect contact with an object by receiving a force toward a body of the stylus. Inputs from the user can be applied to a tip sensor of the stylus via an adapter. The inputs provided via the adapter can be different from inputs applied by contact with an object. Accordingly, the stylus can interpret the user inputs to initiate, terminate, or modify characteristics of the drawing session.

Components of the stylus can be configured to receive input directly from the user to indicate when position measurements are to be taken. For example, the stylus can include an input device, such as a button #150 (seen in patent FIG. 2 below) set within the body #110.

In some embodiments, the button is an input device for receiving an input (e.g., force) from a user. The button can include a switch, knob, lever, force sensor, and/or another input device. Any number of buttons can be provided for receiving user input. The user can operate the button to initiate a usage session, terminate a usage session, or command the stylus to perform other functions. 

The stylus can be used to virtually draw in three-dimensional space. Illustrations can be created in free space by motion of the stylus by the user. One or more aspects of calibration and motion tracking can be applied to track free motion during a drawing session.

The user can operate the adapter #200 and/or the button to initiate a drawing session, in which a marking is virtually created starting at the location of the tip probe #190 in the coordinate system (e.g., in virtual space).

As the user moves the stylus it stylus provides outputs indicating its locations in virtual space during the session (e.g., when an input device is operated with a force). The locations in physical space can correspond to a physical location of the stylus during the session.

Markings continue to be generated along the pathway defined by the locations. When the user provides an indication (e.g., by releasing or pressing the adapter $200 and/or the button #150) to cease the drawing session, the marking ceases to be generated, and the user can move the stylus 100 without generating additional markings.

The recording of markings can be performed independently of whether the tip probe makes contact with a surface. According to some embodiments, the stylus can record motions by the user during a drawing session and transmit the applicable data as an output for later usage.

According to some embodiments, the stylus can communicate with a device for visual display to the user, with which the markings are displayed for the user as they are generated during a drawing session.

For example, the display can include a headset, a head-up display, and/or an optical head-mounted display in communication with the stylus.

By further example, the display can include a flat-panel display that displays a three-dimensional rendering of the markings. Similarly, the stylus provides outputs indicating its locations in virtual space when the tip probe is applied to surface. The locations can then correspond to a physical location of the stylus while the tip probe is applied to the surface.

Apple's granted patent was originally filed in Q4 2 2017 and published today by the US Patent and Trademark Office.