Yesterday evening Patently Apple posted a report titled "Apple's In-Depth Work on a Next-Gen Mixed Reality Headset is Simply Mind Boggling in Scope." It was one of the coolest patent filings Apple has made in some time. At the end of our report I presented a single patent figure from a second related patent that was published yesterday to demonstrate where Apple could take their next generation mixed reality headset. That patent figure is now presented in this follow-up report. The figure was taken from Apple's patent filing titled "3D Document Editing System."

In-part and in reality, Apple's mixed reality headset could double as a portable desktop and/or desktop replacement. For professionals on the road, using a mixed reality headset as their 30" display to work on large documents or architectural drawings or an ad campaign is vastly superior to a notebook to be sure.

Apple notes in their filing that conventional graphical user interfaces (GUIs) for text generation and editing systems work in a two-dimensional (2D) space (e.g., a 2D screen or page on a screen). Highlighting areas or portions of text using these GUIs typically involves adding some effect in 2D such as bold or italics text, underlining, or coloring.

Virtual reality (VR) allows users to experience and/or interact with an immersive artificial three-dimensional (3D) environment. For example, VR systems may display stereoscopic scenes to users in order to create an illusion of depth, and a computer may adjust the scene content in real-time to provide the illusion of the user interacting within the scene.

Similarly, augmented reality (AR) and mixed reality (MR) combine computer generated information with views of the real world to augment, or add content to, a user's view of their environment. The simulated environments of VR and/or the enhanced content of AR/MR may thus be utilized to provide an interactive user experience for multiple applications, such as interacting with virtual training environments, gaming, remotely controlling drones or other mechanical systems, viewing digital media content, interacting with the internet, or the like.

Conventional VR, AR, and MR systems may allow content consumers to view and interact with content in a 3D environment. Conventional VR systems may provide tools and applications that allow VR content creators to create and edit 3D objects, and may provide a text generation and editing system with a conventional 2D GUI that allows content creators to generate text content that can be attached to 3D objects.

However, these conventional VR systems typically do not provide text generation and editing systems with GUIs that allow content creators to generate and edit text with 3D effects in a VR 3D environment. This is what Apple's invention is set to rectify.

Apple's 3D Document Editing System

Apple's invention covers methods and apparatus for generating and editing documents with three-dimensional (3D) effects for text content in a 3D virtual view space. Embodiments of 3D document editing systems, methods, and graphical user interfaces (GUIs) are described that may include a virtual reality (VR) device such as a VR headset, helmet, goggles or glasses for displaying documents in a 3D virtual space, and an input device (e.g., keyboard) for entering and editing text or other content in the documents and that includes sensing technology for detecting gestures by the user.

The VR device and input device may be coupled via a wired or wireless (e.g., Bluetooth) connection. The VR device may be configured to display a 3D text generation and editing GUI in a virtual space that includes a virtual screen for entering or editing text in documents via a keypad of the input device.

Unlike conventional 2D graphical user interfaces, using embodiments of the 3D document editing system, a text area or text field of a document can be placed at or moved to various Z-depths in the 3D virtual space.

A Specialized Keyboard

Apple further notes that the input device (e.g., keyboard) may include sensing technology, for example a motion, touch, and/or pressure sensing region or area on a keyboard, for detecting a user's gestures, for example motions of the user's thumbs when on or near the sensing region.

The VR device may detect gestures made by the user via the sensing technology, and in response may move selected content in a document (e.g., words, paragraphs, sections, columns, sentences, text boxes, uniform resource locators (URLs) or other active text, etc.) forward or backward on a Z axis in the 3D virtual space relative to the rest of the document according to the detected gestures.

The sensing technology of the keyboard may be configured to allow the user to make the gestures while entering text via the keypad, thus allowing the user to provide the 3D effects to text while typing.

In some embodiments, the gestures may include a gesture (e.g., moving both thumbs down on a sensing region of a keyboard) to move an area of the document (e.g., a paragraph, section, column, text field, text box, etc.) forward on the Z axis in 3D space relative to the document.

In some embodiments, the gestures may include a gesture (e.g., moving both thumbs up on the sensing region) to move an area of the document backward on the Z axis in 3D space relative to the document. In some embodiments, the gestures may include a gesture (e.g., moving one thumb down on the sensing region) to move a portion of text in the document (e.g., a uniform resource locator (URL), sentence, word, title or heading, etc.) forward on the Z axis in 3D space relative to other content of the document. In some embodiments, the gestures may include a gesture (e.g., moving one thumb up on the sensing region) to move a portion of text in the document backward relative to other content of the document.

In some embodiments, the VR device may instead or also monitor text entry made using a keypad of the keyboard, apply rules to the text entry to determine particular kinds of text items such as URLs, and automatically shift the particular kinds of text items (e.g., URLs) forward or backward on the Z axis relative to the rest of the content in the document.

In some embodiments, a document generated using the 3D text editing system may be displayed to content consumers in a 3D virtual space via VR devices, with portions of the document (e.g., paragraphs, text boxes, URLs, sentences, words, sections, columns, etc.) shifted backward or forward on the Z axis relative to the rest of the content in the document to highlight or differentiate those parts of the document.

For example, active text fields or hot links such as URLs may be moved forward relative to other content in the document so that they are more visible and easier to access by the consumer in the 3D virtual space using a device such as a controller or hand gestures.

In some embodiments, the VR device may also provide augmented reality (AR) or mixed reality (MR) by combining computer generated information with a view of the user's environment to augment, or add content to, a user's view of the world. In these embodiments, the 3D text generation and editing GUI may be displayed in an AR or MR view of the user's environment.

Below we present patent FIG. 1A that illustrates a 3D document editing system; FIG. 2A shows the text area #212 displayed at the same depth as the document #210 in the 3D virtual space #202. The keyboard #220 may include a sensing area #224 that implements one or more of motion sensing, pressure/force sensing, or capacitive (touch) sensing technology located below the keypad #222 to be easily accessible by user's thumbs. FIG. 2A shows user's left and right thumbs touching at respective contact points #226 in the sensing area.

The patent figures above are illustrating a virtual "editing system" Apple's patent FIG. 1B presented below illustrates a 3D document "viewing system."

Apple's patent FIG. 6 below is a block diagram illustrating components of a 3D document editing system.

Apple's patent application #20180081519 was filed back in Q3 2016. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.