Today the U.S. Patent & Trademark Office published a patent application from Apple that reveals their intent to advance their parallax technology for iOS to the next level by driving parallax depth rendering. This was a cornerstone of yesterday's IDF keynote with CEO Brian Krzanich. Intel's CEO talked up their RealSense camera technology for mobile devices, robot navigation and beyond in context with advancing sensory perceptions for computing. Apple possesses the very same kind of technology that's behind RealSense via their newly acquired technology from PrimeSense and LiNX. It involves head-tracking and 3D depth cameras to advance photography and video apps in the future and much more.

Apple notes that a technique is presented in their patent filing for rendering still images and videos that enable a conventional 2D display to give an observer the appearance that the 2D display is actually a window into a 3D world.

This technique is referred to herein as "depth rendering." The depth rendering is accomplished by simulating the physical imaging characteristics of parallax and depth of field. The depth rendering typically employs three inputs: 1) an input image or video; 2) a way to segment the input image into 2 or more distinct regions; and 3) relative orientation information between the display and the observer.

Depth rendering is a dynamic rendering of an image on a 2D display that changes as the observer adjusts their orientation (spatial-relationship) with the device in real time. By segmenting an image into 2 or more different regions the regions can be parameterized and then numerically altered to simulate the effect of viewing a 3D scene. More specifically, depth rendering employs a simulation of parallax and a simulation of depth of field.

The parameters of an image segment that can be altered to create the parallax effect include (but are not limited to) the position, scale, rotation, perspective and distortion of the image segment. The parameters of an image segment that can be altered to create the depth of field effect include (but are not limited to) the blur, sharpness, scale, position, rotation, color, contrast, saturation, hue and luminance of the image segment. In some embodiments, the depth rendering may modify parameters of less than all of the image segments.

Depth rendering involves rendering the various image segments as separate planes that are then superimposed on top of each other. Conceptually the superimposition can be visualized as a vertical stack of image segments ordered by increasing depth from top to bottom of the stack such that segments near the top of the stack may or may not occlude segments near the bottom of the stack. A given image segment is changed by altering one or more of the parameters of the segment. The change in the segment affects the occlusion of the image segments below the given image segment in the stack, which in turn simulates the effect of parallax and or depth of field. By coupling the parameters of the image segments to the relative observer-display orientation data, the effects of parallax and depth of field can be simulated in real time to create a user experience that mimics viewing a 3D scene.

Apple's patent FIG. 1 noted below is a block diagram illustrating an example of a simulated three-dimensional image at two orientations

Although Apple introduced the parallax effect for the iOS 8 home screen, this patent goes much further as it uses depth technology that they gained via their PrimeSense acquisition.

Apple notes in their filing that "Recently several new technologies have been introduced that enable creating a 2D map of the three-dimensional (3D) coordinates of a scene; for example, the Microsoft Kinect device. ("KINECT" is a registered trademark of Microsoft Corporation.) There are several different technologies at this time that can create such depth maps: time of flight, structured light, spatial phase imaging, polarization-based methods, lidar, radar, etc. Alternatively, there are techniques to create a 3D depth map from a sequence of 2D images, either captured from a single camera or from a collection of cameras, such as computed tomography or more crudely by simply capturing a sequence of 2D images with an adjustable focus camera.

The technology described above relating to 3D depth maps could reviewed in a series of patents from Apple that you can find here. It's a very hot topic of late, with Intel talking about it extensively during their opening IDF Keynote yesterday regarding their RealSense camera that will advance a new generation of sensory experiences. Apple's PrimeSense technology covers the very same ground and far beyond.

So introducing 3D photography and videos apps for future iDevices isn't a matter of if but rather when. Will it be for the 2016 iPhone 7 or will Apple shock the market by introducing something ultra-cool for the iPhone 6s in September?

With the forecasts for the iPhone 6s being so high, you have to wonder if Apple will be introducing a new feature to excite the market at a time when many analysts think that smartphone sales will stagnate or fall in Q4. Will Apple break with tradition and go far beyond a ho-hum iPhone upgrade? Only time will tell.

More about Depth Mapping

Apple's patent FIG. 4 noted above is a photograph of a 3D object posed in front of a background. In this view, the observer is positioned directly in front of the object #410. In Apple's patent FIG. 6 above which is below FIG. 4 we're able to see that the image created using the technique described in the patent filing, has now shifted to the left using a 3D effect.

Apple's patent FIG. 5 noted above is a representation of a depth map #500 created for the 3D object of FIG. 4. Recently several new technologies have been introduced that enable creating a 2D map of the three-dimensional (3D) coordinates of a scene using Microsoft's Kinect.

Using a sequence of images that sweep through different focusing points a coarse model can be generated that separates a subject object from the background of the image. A simple binning of focus groups into near and far may be sufficient for a depth map. A pair of images, one made with a flash illumination and one that is made without the flash illumination shows how the flash illuminates the scene and can be used to separate foreground from background objects.

In certain types of images, such as portraits, where a portion of the image is well isolated from the rest of the image, segmentation may be done based on that portion of the image. In another technique, multiple images with movement of objects in the image can be used to segment the moving objects from the non-moving objects. Any desired technique for creating a depth map may be used.

Furthermore, devices with built in cameras can detect the presence of an observer and track the orientation of the observer with respect to the device by detecting the location and orientation of the observer's face, body, eyes, gaze, gesture, etc. and track in real time how the relative position and orientation of the observer changes.

Patent Credits

Apple credits Kevin Gross, Richard Baer and Damien Thivent as the inventors of this patent application which was originally filed in Q1 2014. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Comments are reviewed daily from 5am to 6pm MST and sporadically over the weekend.