A Future Version of Apple Pencil with a 3-Camera System could be used to map-out 3D-Environments for Next-Gen Glasses & more
Today the US Patent & Trademark Office published a patent application from Apple covering a future Apple Pencil with a 3-camera system designed to assist next Apple Glasses and/or Head Mounted Device (HMD) map out 3-D environments, act as a tracking device and more.
A head-mounted device can be worn by a user to display visual information within the field of view of the user. The head-mounted device can be used as a virtual reality (VR) system, an augmented reality (AR) system, and/or a mixed reality (MR) system. A user may observe outputs provided by the head-mounted device, such as visual information provided on a display. The display can optionally allow a user to observe an environment outside of the head-mounted device. Other outputs provided by the head-mounted device can include speaker output and/or haptic feedback.
A user may further interact with the head-mounted device by providing inputs for processing by one or more components of the head-mounted device. For example, the user can provide tactile inputs, voice commands, and other inputs while the device is mounted to the user's head.
Apple notes that Head-mounted devices that are an attractive technology for a variety of applications, and can be used to provide an immersive and interactive computer-generated reality (CGR). For example, head-mounted devices can be used to provide VR or MR environments that are responsive to movements of a user in three-dimensional space for purposes such as gaming, video entertainment, and training simulations, among other possibilities.
A head-mounted device can be equipped with sensors or motion tracking technology to track movements of the user's head, and generate an appropriate response in output of the head-mounted device (e.g., audio, graphics, etc.).
A system equipped with the ability to track or respond to other movements of the user or environment can provide greater immersion and new possibilities for the user experience.
For example, the ability to track free space movements of the user's hands or other physical objects can allow for three-dimensional manipulation of objects in a CGR environment.
Additionally, or in combination, it would be desirable to provide the user with additional tools with which the user can observe or otherwise interact with the physical environment while wearing a head-mounted device.
According to some embodiments, a system involving a head-mounted device and a future Apple Pencil could be used to track free-space movements of a user.
This Pencil could be equipped with one or more light emitters, and the head-mounted device can be equipped with an outward facing camera that is configured to transduce movements of the Pencil based on detection of the light emitters.
Such a system can provide a self-contained tracking system that allows a user to freely move the Pencil in three-dimensional space, while accurately transducing the free-space movements of the Pencil without a need for external cameras.
Additionally, or in combination, the Pencil could equipped with one or more cameras to facilitate tracking of movements of the stylus or for gathering additional data from a physical environment around the stylus in order to affect output provide by the head-mounted device.
For example, cameras in the Pencil could be configured to provide a view to spaces that are obstructed or hidden from a field of view of a user wearing the head-mounted device, allowing a user to peer around objects in three-dimensional space with a stylus and view occluded or hidden spaces on the head-mounted device.
In some embodiments, multiple cameras may be included in the Pencil, such as one in the body region and one near the tip, so that the Pencil is capable of both observing a three-dimensional environment around the stylus and an interface surface contacted by the tip of the Pencil.
Additionally, or in combination, one or more cameras in the Pencil could be used to track light emitters in the head-mounted device to track a free space position of the Pencil or improve an accuracy of a tracking determination made in combination with other data sources.
Apple's patent FIG. 1 below illustrates a schematic diagram of an example of a system having a head-mounted device and Apple Pencil. This future Apple Pencil could include one or more cameras (#325). The cameras could be configured to optically image a physical environment outside of the Apple Pencil. Such image information could be used to track a position of the Apple Pencil.
For example, the camera on Apple Pencil could be used to track movements in environmental features, and position or movement of Apple Pencil could be determined based on the movements in the environmental features detected with the camera.
In some embodiments, the head-mounted device #200 could be equipped with a light emitter or other visual target, and the camera and Apple Pencil could be used to detect the light emitter of the head-mounted device. Such information could be used alone, or in combination with other collected data, such as the data gathered by tracking the light emitters. Combining such data can enhance the accuracy or increase the confidence of tracking determinations made by the system #100.
Apple's patent FIG. 2 below illustrates a perspective view of a future Apple Pencil; FIG. 3 illustrates a front view of a future Apple Pencil having one or more light emitters; and FIG. 4 illustrates a front view of an example of a future Apple Pencil having a pair of light emitters with different spatial patterns; FIG. 5 illustrates a front view of Apple Pencil having a light emitter with a pattern that varies about a circumference of the Pencil.
Apple's patent FIG. 6 below illustrates a front view of an example of an Apple Pencil having multiple cameras; and lastly FIG. 7 illustrates a block diagram of an example of a system having a head-mounted device and a future Apple Pencil.
Further to FIG. 6, Apple notes that "the second camera #325b may be used to, for example, track motions of the Pencil as it is moved along or across the surface of a device" that appears to be an iPhone.
Such a camera may track changes seen in the texture of the interface surface of the portable device to determine motion of the Pencil, for example, to implement writing or drawing functionality in a mixed reality environment beyond a single device and across substantially any surface.
Apple's patent application number 20200333891 that was published today by the U.S. Patent Office was filed back Q1 2020. Considering that this is a patent application, the timing of such a product to market is unknown at this time.