Apple wins a Patent for an HMD and/or Smartglasses interacting with a Virtual Keyboard that works with a Tracking Device
Today the U.S. Patent and Trademark Office officially granted Apple a patent that relates to a head-mounted device that can provide a user experience that is immersive while allowing the user to interact with the system in a manner that is natural, familiar, and intuitive. The patent covers using a physical or virtual keyboard and Apple Pencil to draw using a tracking device accessory. The virtual keyboard could provide users with many little advantages over a physical keyboard.
While a head-mounted device can provide outputs to a user in a variety of ways, it can also be helpful to allow a head-mounted device to receive inputs from a user. A head-mounted device can provide a user experience that is immersive while allowing the user to interact with the system in a manner that is natural, familiar, and intuitive.
Head Mounted Device + Input via Keyboard
Apple notes that a keyboard is an example of an input device that is familiar to many computer users. The use of a head-mounted device can keep a user's hands free to operate a keyboard or another device in a manner that is similar to the use of a keyboard.
The operation of a keyboard, another input device, and/or any surface can be enhanced by features of the head-mounted device, which can display feedback, outputs, or other features based on the use of the keyboard.
For example, the head-mounted device can display text generated by the user's operation of the keyboard. The text can be displayed in a manner that allows a user to readily see the keyboard, the user's hands, and the text that is generated by operation of the keyboard.
The head-mounted device can further display features that facilitate the user's operation of the keyboard. For example, suggested text, keystrokes, or other features correlated with keys of the keyboard can be displayed for selection by a user.
By further example, the keyboard can be displayed in a position and orientation that conforms to an arrangement of the user's hands within a field of view of the head-mounted device.
Apple's patent FIG. 1 below illustrates a view of a head-mounted device and a keyboard in use; FIG. 2 illustrates a view of a head-mounted device and an input device in use; FIG. 3 illustrates a view of a head-mounted device and a surface in use.
As shown in FIG. 1, the tracking device (#200) can be placed in close proximity to the keyboard (#300) to track motion of the user's hands and/or the keys of the keyboard during operation. The tracking device can be in communication with the head-mounted device such that detections by the tracking device can generate signals that are transmitted from the tracking device to the head-mounted device.
In FIG. 2 above, the head-mounted device can be used in conjunction with another input device, such as input device #400 which could be a virtual keyboard and the HMD provides the visuals of a keyboard. Between the cameras in the HMD along with a tracking device, the user's input could be accurately understood as if it were a physical keyboard.
In FIG. 3, Apple takes it one step further in that a user will be able to use a desk, table, wall to not only use a keyboard but be able to write or draw with an Apple Pencil.
Apple's patent FIG. 7 below illustrates a display of a head-mounted device providing a view of a keyboard, a user's hands, and text within a window; FIG. 8 illustrates a display of a head-mounted device providing a view of a keyboard, a user's hands, and text with available selections.
Apple's patent FIG. 11 below illustrates a display of a head-mounted device providing a view of a keyboard, a user's hands, and virtual indicators; and FIG. 14 illustrates a display of a head-mounted device providing a view of keyboard portions having an arrangement based on a position of a user's hands.
For more details, review Apple's granted patent 11,137,908.