It appears that certain Apple engineering teams are working on new forms of Apple Pencil that could work with the current or future versions of Apple Vision Pro. On January 11^th, we posted a report titled "Apple invents a new kind of Hand Controller that is designed to work with HMDs …" Today, the U.S. Patent and Trademark Office officially granted Apple a patent that relates to a possible new or upgraded Apple Pencil specifically designed to work with an HMD like Apple Vision Pro. It could be used for gaming, training simulations and more. With cameras on the stylus, a user using Apple Vision Pro or other HMD to see around 3D objects and more. 

Stylus-based Input System For A Head-mounted Device

Head-mounted devices are an attractive technology for a variety of applications, and can be used to provide an immersive and interactive computer generated reality (CGR). For example, head-mounted devices can be used to provide VR or MR environments that are responsive to movements of a user in three-dimensional space for purposes such as gaming, video entertainment, and training simulations, among other possibilities. A head-mounted device can be equipped with sensors or motion tracking technology to track movements of the user's head, and generate an appropriate response in output of the head-mounted device (e.g., audio, graphics, etc.).

A system equipped with the ability to track or respond to other movements of the user or environment can provide greater immersion and new possibilities for the user experience. For example, the ability to track free space movements of the user's hands or other physical objects can allow for three-dimensional manipulation of objects in a CGR environment. Additionally, or in combination, it would be desirable to provide the user with additional tools with which the user can observe or otherwise interact with the physical environment while wearing a head-mounted device.

According to some embodiments disclosed herein, a system involving a head-mounted device and a handheld stylus can be used to track free-space movements of a user. The stylus can be equipped with one or more light emitters, and the head-mounted device can be equipped with an outward facing camera that is configured to transduce movements of the stylus based on detection of the light emitters. Such a system can provide a self-contained tracking system that allows a user to freely move the handheld stylus in three-dimensional space, while accurately transducing the free-space movements of the stylus without a need for external cameras.

Additionally, or in combination, the stylus can be equipped with one or more cameras to facilitate tracking of movements of the stylus or for gathering additional data from a physical environment around the stylus in order to affect output provide by the head-mounted device.

For example, cameras in a stylus configured to provide a view to spaces that are obstructed or hidden from a field of view of a user wearing the head-mounted device, allowing a user to peer around objects in three-dimensional space with a stylus and view occluded or hidden spaces on the head-mounted device.

In some embodiments, multiple cameras may be included in the stylus, such as one in the body region and one near the tip, so that the stylus is capable of both observing a three-dimensional environment around the stylus and an interface surface contacted by the tip of the stylus.

Additionally, or in combination, one or more cameras in the stylus can be used to track light emitters in the head-mounted device to track a free space position of the stylus or improve an accuracy of a tracking determination made in combination with other data sources.

Apple's patent FIG. 1 below illustrates a schematic diagram of an example of a system having a head-mounted device and a stylus/Apple Pencil.  

the stylus #300 may be configured as a wireless device that can be freely moved in three-dimensional space. the camera #325 may be used to collect data or image spaces that are hidden or occluded from a field of view of the user #10 and/or the sensor #240 of the head-mounted device #200. Such data can be used to map a topography of the physical environment within which the electronic system #100 is operated. For example, the mapping may be used to affect how virtual objects of a mixed reality environment interact with the physical environment.

Images of the environment captured with the camera of the stylus can be directly presented on the display #220 of the head-mounted device to permit the user to peer around or inside of objects in the physical environment. Such images can include still images and/or images form a video. 

In some embodiments, the video captured with the camera of the stylus  can be presented on the head-mounted device in real time to permit the user to view hidden or occluded spaces on the head-mounted device using the stylus like a periscope.

Additionally, or in combination, the stylus and/or the head-mounted device can include one or more other sensing components, which can include non-optical sensing components such as, for example, ultrasonic sensing components and/or electromagnetic sensing components.

Apple's patent FIG. 2 above illustrates a perspective view of an example of a stylus. The stylus may further include a control input #358 which is designed to receive commands separate from touch inputs provided by the tip #390. For example, the control input can include a button, touch sensor, microphone, or other input component capable of receiving inputs from the hand or other part of the user.

The control input can be used to activate components of the stylus, change operating modes, change settings, or provide other functionality in support of stylus-based input to the electronic system #100.

For example, the control input can be used to power up or power down the stylus; activate the light emitter #315; activate the camera #325;  activate a shutter of the camera; trigger selection, navigation, or gaming inputs in a CGR presented on the head-mounted device 200.

Apple's patent FIG. 3 above illustrates a front view of an example of a stylus having one or more light emitters; FIG. 4 illustrates a front view of an example of a stylus having a pair of light emitters with different spatial patterns; FIG. 5 illustrates a front view of an example of a stylus having a light emitter with a pattern that varies about a circumference of the stylus; FIG. 6 illustrates a front view of an example of a stylus having multiple cameras.

Apple's patent FIG. 7 below illustrates a block diagram of an example of a system having a head-mounted device and a stylus.

To cover their bases, Apple notes that while the main focus of the patent relates to a stylus/Apple Pencil is for an HMD like Vision Pro, this new Apple Pencil may one day work with the iPhone, Macs, iPad and more.

For more details, review Apple's granted patent 11886650.