While Apple's CEO told the UK's Independent yesterday that Augmented Realty 'Glasses' are years away. AR glasses are a different type of invention and not to be confused with a Virtual Reality Headset that could be used for a different kind of experience. Apple hired Jeff Norris earlier this year who worked at NASA's Jet Propulsion Lab where he worked on both Virtual and Augmented Reality projects. His team worked with Facebook's Oculus (Virtual Reality), Microsoft's HoloLens (Mixed Reality) and the HTC Vive which Apple promotes on their iMac web page as noted below. Apple states that the new iMacs have the power to create cutting-edge VR content for 3D VR headsets like the HTC Vive.

Today the US Patent & Trademark Office published a patent application from Apple that relates to monoscopic (1 view) or stereoscopic (2 view) 180 degree (hemisphere) or 360 degree (full sphere) photography and post-production for distribution to dome, mobile or head-mounted display virtual reality applications.

Apple's Patent Background

Conventional 180 degree or 360 degree video and/or images are stored in flat storage formats using equirectangular or cubic projections to represent spherical space. If these videos and/or images are edited in conventional editing or graphics applications, it is difficult for the user to interpret the experience of the final result when the video or images are distributed and presented in a dome projection, cubic, or mapped spherically within a virtual reality head-mounted display. Editing and manipulating such images in these flat projections requires special skill and much trial and error.

Further, it is not an uncommon experience to realize after manipulation of images or videos composited or edited with spherical that subsequent shots are misaligned, or stereoscopic parallax points do not match in a natural way.

Apple's Invention

Apple's invention is generally directed to methods and systems for transmitting monoscopic or stereoscopic 180 degree or 360 degree still or video images from a host editing or visual effects software program as equirectangular projection, or other spherical projection, to the input of a simultaneously running software program on the same device that can continuously acquire the orientation and position data from a wired or wirelessly connected head-mounted display's orientation sensors, and simultaneously render a representative monoscopic or stereoscopic view of that orientation to the head mounted display, in real time.

In one embodiment, a method for presenting an image preview is described. The method may include receiving image data from a GPU buffer, wherein the GPU buffer is associated with a media manipulation application, obtaining an orientation data for a display device, rendering a preview image using the image data and the orientation data, and displaying the preview image using the display device, wherein the preview image is displayed while the image data is utilized by the media manipulation application, and wherein the preview image is dynamically modified when the image data is modified by the media manipulation application.

Apple's patent FIG. 5 shown above illustrates an example for presenting a preview image by a 3D display device. FIG. 5 illustrates the user interface of a video or image editing or graphics manipulation software program #501 with an equirectangularly projected spherical image displayed in the canvas #502 and a compositing or editing timeline #503.

The image output of the video or image editing or graphics manipulation software program can be output via a video output processing software plugin module #504 and passed to a GPU image buffer shared memory and then passed efficiently to the image receiver #507 of the head-mounted display processing program #506.

The 3D image processing routine #508 of the head-mounted display processing program will texture the inside of a virtual sphere or cube with a 3D viewpoint at the center of said sphere or cube. The virtual view for each of the left and right eyes will be accordingly cropped, duplicated (if necessary), distorted and oriented based on the lens/display specifications and received orientation data #509 of the wired or wirelessly connected head-mounted display's #510 orientation sensor data. Once the prepared image is rendered by the 3D image processing routine, the image can then be passed to the connected head-mounted display for immediate presentation to the wearer within the head-mounted display.

Additionally, since wearing a head-mounted display will obscure the wearer's view of the UI of the video or image editing or graphics manipulation software program, it is also possible to capture the computer display's user interface as an image using a screen image capture software program module #512 and pass it to an image receiver/processor #513 for cropping an scaling before being composited on the left and right eye renders from the 3D image processing routine #508, #514, #515 and then the composited image can be passed to the connected head-mounted display #511 for immediate presentation to the wearer within the head-mounted display.

Further, a redundant view can be displayed in a window #516 on the computer's display so others can see what the wearer of the head-mounted display is seeing, or if a head-mounted display is not available.

According to one or more embodiments, the benefits of this system and/or method in accordance with one or more embodiments can include a dramatic decrease in the amount of time required to align, manipulate, or edit spherical 360 degree video or images. Though embodiments of the present invention are discussed with reference to image data, embodiments of the present invention also apply to audio data.

A description of common computing software architecture is provided and expressed in a layer diagram in FIG. 2. Like the hardware examples, the software architecture discussed here is not intended to be exclusive in any way, but rather to be illustrative. This is especially true for layer-type diagrams, which software developers tend to express in somewhat differing ways.

Apple states in their patent filing that "it may be useful to note that Metal #252 represents a published framework/library of Apple Inc. that is known to developers in the art. Furthermore, OpenGL #251 may represent a framework/library present in versions of software either currently or formerly distributed by Apple Inc.

Apple further notes that "Sprite Kit #261 is a graphics rendering and animation infrastructure made available by Apple Inc. Sprite Kit may be used to animate textured images or "sprites." Scene Kit #262 is a 3D-rendering framework from Apple Inc. that supports the import, manipulation, and rendering of 3D assets at a higher level than frameworks having similar capabilities, such as OpenGL. Core Animation #263 is a graphics rendering and animation infrastructure made available from Apple Inc. Core Animation may be used to animate views and other visual elements of an application. Core Graphics #264 is a two-dimensional drawing engine from Apple Inc. Core Graphics #265 provides 2D rendering for applications.

Apple's patent FIG. 1 illustrates a simple flow chart noting that electronic device #100 may be part of a multifunctional device, such as a mobile phone, laptop computer, tablet computer, personal digital assistant, portable music/video player, or any other similar electronic device. According to one or more embodiments, the electronic device may be coupled to a 3D display system #160, either wired or wirelessly.

Apple's patent FIG. 3 presented below illustrates a flowchart form, a method for presenting a preview image by a 3D display device.

Apple's patent application 20170295361 was filed back in Q2 2017. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.