Six Apple ‘Finger Device' patents have been published in 6 weeks with today’s being focused on finger devices for large virtual keyboards
While Apple's 'Standby' Patent was fulfilled in iOS 17, Apple envisions this feature expanding to special iPhone cases for sports, games & more

The U.S. Patent Office has published Apple's Patent Applications for 'Magical' Spatial Videos and Photos

1 cover Apple patents for Spatial Video & Photos(Click on image to Enlarge)

Today the US Patent & Trademark Office published two patent applications from Apple that relate to both Spatial Videos and Spatial Photos on Vision Pro with content that could also be created on iPhone 15 Pro (later this year)  

Patent One: "Extended Reality Recorder"

Apple's patent covers implementations of the subject technology that provides systems and methods for recording an extended reality experience in a way that allows the experience to be played back at a later time from a different viewpoint or perspective.

The technology covers an XR system that provides recording, storage, editing, management, and playback of extended reality (XR) recordings to facilitate re-viewing of an XR experience from a different viewpoint in the XR environment. In an example, time-ordered three-dimensional (3D) information (e.g., depth information from one or more depth sensors) about the physical environment in which a device is located is captured over a period of time for extended reality recording of an experience.

Over substantially the same period of time, video and/or audio content (e.g., including spatial audio content) can be captured and/or displayed by the device. Over substantially the same period of time, computer-generated (CG) content is generated and displayed over a portion of a view of the environment. CG content corresponding to the CG content that is displayed during the period of time is also captured as time-ordered CG content, and stored aligned in time with the time-ordered three-dimensional (3D) information (e.g., including depth information), metadata including information about the scene, the video and/or the audio.

The CG content corresponding to the CG content that is displayed during the period of time, and that is captured by the XR system, can include a scene graph, the 3D geometries, textures, materials, animations etc. of CG content that is displayed during the period of time, and/or 3D geometries, textures, materials, animations etc. of CG content that would have been displayed during the period of time if the recording user had been looking in other directions or locations in the scene during the period of time, so that the XR recording can be played back from any point of view.

During the period of time of recording (e.g., live, or online) and/or after the recording (e.g., offline), a three-dimensional reconstruction of the physical environment can also be generated (based on the depth information, the video, the audio, and/or other sensor data), the 3D reconstruction can be stored, and, in some examples, a time-ordered reconstruction track can also be stored, aligned in time with the time-ordered CG content, the time-ordered three-dimensional (3D) information, the metadata, the video and/or the audio.

The system is primarily designed for Apple Vision Pro, though the iPhone 15 Pro will allow a user to shoot "Spatial Videos" later this year, according to Apple's iPhone 15 marketing material. These videos will be able to play back on Vision Pro when it debuts in 2024.

2 clip from iPhone 15 Pro marketing page

Apple also interestingly hints that live 3D video may be able to be presented on a vehicle's windshield.

Apple's patent FIG. 2 below illustrates an example software architecture that may be implemented; FIG. 3 illustrates additional features of an electronic device in accordance with implementations of the subject technology.

3. Apple Vision Pro patents for 3d Spatial videos and photos +

Apple's patent FIG. 6 above illustrates example features of a recording of an extended reality experience; Apple's patent FIG. 5 below illustrates various stages of recording, advanced editing, and playing back an extended reality experience. The advanced editing illustrates how a user could add a virtual bear into a scene filmed in a forest. That kind of easy XR editing may be a lot of fun in the future.

(Click on image to Enlarge) 4. ADVANCED XR EDITING - EXAMPLE ADDING A VIRTUAL BEAR TO SCENE

To review the details of this in-depth filing, review patent application 20230298282.

Patent Two: 3D Photos

Apple's second patent covers devices, systems, and methods that determine how to present a three-dimensional (3D) photo in an extended reality (XR) environment (e.g., in 3D, 2D, blurry, or not at all) based on viewing position of a user active in the XR environment relative to a placement of the 3D photo in the XR environment.

In some implementations, at an electronic device having a processor, a 3D photo that is an incomplete 3D representation created based on one or more images captured by an image capture device is obtained. In some implementations, a viewing position of the electronic device relative to a placement position of the 3D photo is determined, and a presentation mode for the 3D photo is determined based on the viewing position.

In some implementations, the 3D photo is provided at the placement position based on the presentation mode in the XR environment.

5 - fig. 2 3-D photos for XR

To review the in-depth details of this invention, review patent application 20230298278.

10.51FX - Patent Application Bar

Comments

The comments to this entry are closed.