Apple Invents multiple ways that a future HMD or Smartglasses could Interact with a Virtual Keyboard & more
Today the US Patent & Trademark Office published a patent application from Apple that relates to a head-mounted device that can provide a user experience that is immersive while allowing the user to interact with the system in a manner that is natural, familiar, and intuitive. The patent covers using a physical or virtual keyboard and Apple Pencil to draw using a tracking device accessory. The virtual keyboard could provide users with many little advantages over a physical keyboard.
A head-mounted device can be worn by a user to display visual information within the field of view of the user. The head-mounted device can be used as a virtual reality (VR) system, an augmented reality (AR) system, and/or a mixed reality (MR) system. A user may observe outputs provided by the head-mounted device, such as visual information provided on a display. The display can optionally allow a user to observe an environment outside of the head-mounted device.
Other outputs provided by the head-mounted device can include audio output and/or haptic feedback. A user may further interact with the head-mounted device by providing inputs for processing by one or more components of the head-mounted device. For example, the user can provide tactile inputs, voice commands, and other inputs while the device is mounted to the user's head.
Head-mounted devices, such as head-mounted displays, headsets, visors, smartglasses, head-up display, etc., can perform a range of functions that are managed by the components (e.g., sensors, circuitry, and other hardware) included with the wearable device.
While a head-mounted device can provide outputs to a user in a variety of ways, it can also be helpful to allow a head-mounted device to receive inputs from a user. A head-mounted device can provide a user experience that is immersive while allowing the user to interact with the system in a manner that is natural, familiar, and intuitive.
Head Mounted Device + Input via Keyboard
Apple notes that a keyboard is an example of an input device that is familiar to many computer users. The use of a head-mounted device can keep a user's hands free to operate a keyboard or another device in a manner that is similar to the use of a keyboard.
The operation of a keyboard, another input device, and/or any surface can be enhanced by features of the head-mounted device, which can display feedback, outputs, or other features based on the use of the keyboard.
For example, the head-mounted device can display text generated by the user's operation of the keyboard. The text can be displayed in a manner that allows a user to readily see the keyboard, the user's hands, and the text that is generated by operation of the keyboard.
The head-mounted device can further display features that facilitate the user's operation of the keyboard. For example, suggested text, keystrokes, or other features correlated with keys of the keyboard can be displayed for selection by a user.
By further example, the keyboard can be displayed in a position and orientation that conforms to an arrangement of the user's hands within a field of view of the head-mounted device.
Apple patent FIG. 1 below1 illustrates a view of a head-mounted device and a keyboard in use; FIG. 2 illustrates a view of a head-mounted device and an input device in use; FIG. 3 illustrates a view of a head-mounted device and a surface in use.
As shown in FIG. 1, the tracking device (#200) can be placed in close proximity to the keyboard (#300) to track motion of the user's hands and/or the keys of the keyboard during operation. The tracking device can be in communication with the head-mounted device such that detections by the tracking device can generate signals that are transmitted from the tracking device to the head-mounted device.
In FIG. 2 above, the head-mounted device can be used in conjunction with another input device, such as input device #400 which could be a virtual keyboard and the HMD provides the visuals of a keyboard. Between the cameras in the HMD along with a tracking device, the user's input could be accurately understood as if it were a physical keyboard.
In FIG. 3, Apple takes it one step further in that a user will be able to use a desk, table, wall to not only use a keyboard but be able to write or draw with an Apple Pencil.
Apple's patent FIG. 7 below illustrates a display of a head-mounted device providing a view of a keyboard, a user's hands, and text within a window; FIG. 8 illustrates a display of a head-mounted device providing a view of a keyboard, a user's hands, and text with available selections; FIG. 11 illustrates a display of a head-mounted device providing a view of a keyboard, a user's hands, and virtual indicators; and FIG. 14 illustrates a display of a head-mounted device providing a view of keyboard portions having an arrangement based on a position of a user's hands.
In general, the patent points to the HMD including speakers and a series of sensors that include any of the following: touch, force, temperature, position, motion, and so on. For example, the sensor can be a photodetector, a temperature sensor, a light or optical sensor, an atmospheric pressure sensor, a humidity sensor, a magnet, a gyroscope, an accelerometer, a chemical sensor, an ozone sensor, a particulate count sensor, and so on. By further example, the sensor can be a bio-sensor for tracking biometric characteristics, such as health and activity metrics.
Apple's patent application number 20200326847 that was published today by the U.S. Patent Office was filed back Q1 2020 with some work dating back a year. Considering that this is a patent application, the timing of such a product to market is unknown at this time.