Almost a year ago today Patently Apple posted a report titled "Apple's Mixed Reality Headset Part 2: A 3D Document Editing and Viewing System." Beyond VR gaming and other forms of entertainment, Apple's engineering team envisioned using mixed reality headset for work with a VR keyboard and word processor that could edit documents as our cover graphic from that patent illustrates. That patent was a great overview of a future headset  for school or business.

In today's granted patent that never surfaced as a patent application under Apple, focuses in on specialized VR keyboards designed to work with head mounted display devices. Because it was never published as a patent application publicly we'll cover this as a patent application by covering more details.

In Apple's patent background they note that computers with displays can present simulated environments to users. Head-mounted displays are available that enhance the immersive nature of simulated environments. A user with a head-mounted display may be presented with virtual reality worlds. A user may manipulate virtual objects in the virtual worlds by supplying input through data gloves or other input devices.

Augmented reality is a type of virtual reality in which simulated computer content is merged with the real world. Augmented reality systems may, for example, use head-mounted displays to display virtual objects intermingled with real-world objects.

Although a wide variety of virtual input devices can be constructed for a user in a simulated environment, tasks such as supplying text input can be cumbersome in simulated environments. For example, a user in a simulated environment may not be provided with physical feedback of the type that naturally arises when interacting with a physical keyboard.

Using an Advanced Virtual Keyboard when Wearing a Head Mounted Display Device

Apple's granted patent covers a system in which input is gathered from a user with input-output devices. The input-output devices may include physical keyboards, touch screen displays, data gloves, proximity sensors, cameras, and other input-output devices.

A user may supply text input for a word processing program or other application by typing on a physical keyboard, by supplying touch input to a touch screen display, or by supplying other input to the input-output devices.

Control circuitry in the system may use a head-mounted display or other suitable display to display virtual reality content for the user. The virtual reality content may include a virtual keyboard with virtual key labels. The virtual keyboard may be overlaid on top of a physical keyboard so that each virtual key overlaps a respective physical key in the keyboard.

The user may type on the physical keyboard while observing the virtual key labels. Virtual key labels may be updated in response to user key press input and other input.

For example, if a user presses on a letter key on the keyboard, a set of corresponding accented letter options may be displayed as virtual key labels. The user may then press on a one of the accented letter options to select a desired accented letter.

This is presented in Apple's patent FIG. 13 below which is a diagram showing how a keyboard may have a touch screen display that presents accented letter options and other context-dependent options as a user supplies input to a physical keyboard

Apple further notes that the system may include a display such as a touch screen display that forms part of the physical keyboard, may include touch screen displays in tablet computers and other electronic devices, may include cameras and other equipment for monitoring a user's gaze and finger motions, and/or may include other input and output devices.

A touch screen display on a physical keyboard may be used to display dynamically reconfigurable touch screen options. These options may include, for example, accented letter options corresponding to a pressed letter key on the physical keyboard.

The head-mounted display may be used to overlay virtual key labels onto a touch screen display on a physical keyboard or onto a touch screen display in a tablet computer or other electronic device.

Touch screen displays may display content that does not reveal virtual text and other current virtual reality content being viewed by the user with the head-mounted display. For example, a user may supply text to a touch screen display. The text may be incorporated into a virtual document that is displayed for the user with the head-mounted display while the touch screen display is displaying blank content or other content that does not reveal the text.

Apple's patent FIG. 1 below is a schematic diagram of an illustrative virtual reality system.

Apple's patent FIG. 2 below is a side view of an illustrative keyboard having a proximity sensor; FIG. 3 is a side view of an illustrative keyboard having showing how keys may be provided with key switches or other components to gather key press input from a user; FIG. 4 is a side view of a portion of a user's hand showing how data gloves can be used to gather user input.

Apple's patent FIG. 5 above is a cross-sectional side view of an illustrative keyboard having keys that are overlapped by a sensor such as a touch sensor; FIG. 6 is a side view of an illustrative camera of the type that may be used to perform gaze detection operations and gesture recognition operations; and FIG. 7 is a cross-sectional side view of an illustrative touch screen display in a tablet computer or other electronic device showing how a touch sensor in the touch screen display may be used in gathering user input.

Apple's patent FIG. 16 below is a flow chart of illustrative operations that may be performed when gathering touch screen input in a virtual reality system

Apple's granted patent was originally filed in Q2 2017 and published today by the US Patent and Trademark Office.