Last month Patently Apple posted a report  titled "China's largest video game publisher has bowed out of their 'Metaverse' project just as Apple is preparing to introduce their first XR Headset." In that report I pointed to Bloomberg's Mark Gurman reasonably predicting Apple's XR Headset would arrive at this year's WWDC event so as to debut its Apple Reality software kit and Mac-based simulator. In that same report I pointed to a Samsung presentation of being able to access a PC with their future AR glasses along with link to a 2017 Apple patent illustrating a user editing a 3D document within it XR headset using a Keyboard-like device or a virtual keyboard, as presented below.

On February 22, 2023, the European Patent Office published a patent application from Apple titled "Multi-Device Continuity for use with Extended Reality (XR) Systems."  It describes how a user with an iDevice (iPhone/iPad) or Mac (desktop of laptop) could transfer a document in-progress to an XR Headset display to allow a user to edit or finish said document in Extended Reality. The title of the patent suggests and the patent confirms that it's simply describing Continuity extending through to Apple's headset device.

Apple's patent FIG. 2 below illustrates a view of a display of an XR HMD with an iPhone device on with its UI visible; FIG. 4 & 5 illustrates the processor of the HMD transferring the iPhone UI to the display in HMD. A document or webpage on the iPhone UI could be transferred to the display of the HMD that creates the "Continuity" feature extending to future headsets. The user could then work on the document or continue to read a document in the HMD that could create a much larger viewing space.

For patent FIG. 4 above, Apple explains that the UI #302 on the XR display has been moved (e.g., by a gesture input from the user and/or a gaze-based input from the user to the XR Headset device #105) away from the location of the iPhone #110. In the example of FIG. 4, control of the UI and/or the content therein has been completely transferred to the XR Headset.

In the example of FIG. 4, the iPhone's display #208 no longer displays the UI #202 after the XR Headset assumes control. For example, the iPhone's display may be powered off or changed to a low power state.

In the example of a word processing UI, the UI #302 that is generated by XR device, allows the user to continue editing the word processing document that was being edited originally in the iPhone's UI. After transfer of control to XR HMD UI, the content could be modified from a small iPhone-sized UI to an expanded size, moved to a more convenient location in the XR environment, broken out into multiple (e.g., 3D distributed portions), etc.

So, how will the user continue to work on a document in an XR device? According to Apple, new inputs from the user (e.g., to continue editing a document therein or controlling a media player therein) can be provided to electronic device via detection of the user's finger movements (Air-Typing), other gestures, voice inputs, or eye-tracking inputs with camera(s) and/or sensors), such as depending on whether the handoff has been completed. Elsewhere Apple describes a virtual keyboard that could be seen in the XR Headset.

In practice, the patent is clearly about bringing continuity to Apple's future XR Device. The patent however, is very complex in its detailing. To explore Apple's European patent application number EP4136522, click here.

Apple Inventors

• Mike Buerli: AR/VR Software Engineering Manager
• Mark Ebbole: AR/VR Software Engineer, Technology Development Group
• Andy Richardson: Software engineer
• Sam Iglesias: Design and Prototyping
• Tyler Calderone: iOS Applications Engineer