On May 21, 2015, the U.S. Patent & Trademark Office published a patent application from Apple that generally relates to computing devices, and more specifically to computing devices capable of displaying a spatially interactive, combined two-dimensional and three-dimensional display. Two different users viewing content on this kind of display on a computer, TV (other computing device) or iDevice will be able to see different imagery from different angles. Content seen by one user may not be seen by the other looking at the same display at the same time. 

Apple's Patent Background

Despite the availability of various forms of three-dimensional (3D) display technology, 3D displays are not particularly common. In many cases, the lowering of display resolution necessary to implement 3D or the requirement of 3D glasses in order to perceive the 3D may frustrate users to the point that the users prefer not to utilize 3D technology. In other cases 3D implementations may provide higher resolution 3D and/or enable 3D without 3D glasses, but may still not be very user-friendly due to inflexible and/or narrow `sweet spots` (i.e., viewing perspective required in order for 3D to be seen), restriction of displayed 3D to a particular display orientation, ability to display in only 3D or only either in 3D or two-dimensional (2D), and other such issues. Common adoption of 3D displays may not occur without implementation of 3D display technology that is more user-friendly.

Invention: Spatially Interactive Computer Device

Apple's invention relates to systems and methods for displaying a combined 2D and 3D image. A computing device may include a display with an overlay layer that enables the display to present 2D, 3D images, a simultaneous combination of 2D and 3D images, multiple view images (i.e., different users see different images when looking at the same screen), and/or other combinations.

In some implementations, the overlay layer may be one or more liquid crystal display (LCD) matrix pixel masks, a number of lenses, one or more LCD layers configurable as lenses, or various combinations thereof.

In various implementations, the overlay layer may be adjusted to continue display (or alter display) of 3D portions and/or multiple view portions when the orientation of the computing device is changed.

In one or more implementations, the computing device may adjust the overlay layer based on movement and/or position of one or more users and/or one or more eyes of the user(s) in order to maintain the type of image being displayed (2D, 3D, combined 2D and 3D, multiple view, and so on). The computing device may determine and/or estimate such eye movement and/or position utilizing one or more image sensors, one or more motions sensors, and/or other components.

In some implementations, the computing device may be capable of capturing one or more 3D images, such as 3D still images, 3D video, and so on utilizing one or more image sensors. In such implementations, the computing device may utilize a variety of different 3D imaging techniques to capture 3D images utilizing the image sensor(s).

Though the computing device is illustrated as a tablet computing device in some embodiments, it is understood that this is for the purposes of example. In various implementations, the computing device may be any kind of computing device, such as a handheld computer, a mobile computer, a tablet computer, a desktop computer, a laptop computer, a personal digital assistant, a cellular telephone, a smart phone, and/or any other computing device.

Apple notes that multiple users may be tracked by and view the display system. The display system may determine and track the location of one or more users' eyes and/or gazes in order to accurately present 2D, 3D, or combination 2D/3D images to the users, as well as to update such images in response to motion of the user and/or the system. Thus, two users standing side by side or near one another may see the same points on the display, or the blocking/mask layers may activate to show each user a different image on the display.

3D Glasses

Apple notes in one scenario that if both users are wearing polarized glasses or shutter glasses such as the type commonly used to display 3D images on current conventional 3D devices, each user will be able to see a different 3D image on the display.

The 3D glasses could operate with the display to generate a first 3D image on the first display layer, which may be seen by the first user but blocked from the second user's view by the mask layer (e.g., blocking points). The second user may view a 3D image generated on the second layer in cooperation with the second user's 3D glasses, while this second image is blocked from sight of the first user by the mask layer. Thus, each of the two display layers may be capable of either displaying polarized images to cooperate with appropriately polarized glasses, thereby generating a 3D image for a wearer/user, or may be capable of rapidly switching the displayed image to match the timing of shutters incorporated into the lenses of the glasses, thereby presenting to each eye of the wearer an at least slightly different image that cooperate to form a 3D image.

As illustrated in FIG. 7, first (#703) and second users (#704) are both able to view an iPad display screen. Although both users are able to view the display screen, the iPad may be able to operate in a multiple view mode (or dual view mode, since there are two users) based on the respective viewing perspectives of the first and second users.

Such a multiple view may provide a different version of the display screen to each user or provide respective versions to the respective users that include some common portions and some individual portions as is presented in the FIGS. 7B-C below. Note that one player will see one set of cards (7B) on the left while the other user will only see his cards on the right (7C).

Apple's patent FIG. 2C illustrates all of the liquid crystal elements of one or more of the LCD matrix mask layers being activated and/or deactivated collectively. Activation and/or deactivation of a particular liquid crystal element may be accomplished by subjecting a portion of a particular LCD matrix mask layer to an electrical field.

In FIG. 2C, the liquid crystal elements (#205) of the second layer (#204) are activated, blocking points #214 and #213. As such, the user's left eye #212 sees point #216 and the user's right eye sees point #215. In this way, the LCD matrix mask layers may be utilized to control which portions (such as pixels, pixel elements, and so on) of the display layer are visible to the respective eyes of one or more users.

In addition, multiple users may be tracked by and view the system. The system may determine and track the location of one or more users' eyes and/or gazes in order to accurately present 2D, 3D, or combination 2D/3D images to the users, as well as to update such images in response to motion of the user and/or the system. Thus, two users standing side by side or near one another may see the same points on the display, or the blocking/mask layers may activate to show each user a different image on the display.

It should be noted that Apple may have found a way around having to use 3D glasses, as the users noted in patent figures 2C and 7 are not wearing glasses and in fact, there are no illustrations covering 3D Glasses. Additionally, Apple notes in their patent background that there are techniques to present 3D imagery without glasses. So the 3D glasses that Apple describes above may be used to enhance the 3D experience.

3D Map Feature: Next-Gen Version of Flyover

Lastly, one aspect of this patent application could be considered in-play with Apple's "Flyover" Maps feature. The patent filing notes that "As one non-limiting example, a three-dimensional map of a city or other region may be generated. The system may track the relative orientation of the user with respect to the system. Thus, as the relative orientation changes, the portion, side or angle of the map seen by the user may change. Accordingly, as a user moves the system, different portions of the three-dimensional map may be seen.

This may permit a user to rotate the system or walk around the system and see different sides of buildings in the city, for example. As one non-limiting example, this may permit a map to update and change its three-dimensional display as a user holding the system changes his or her position or orientation, such that the map reflects what the user sees in front of him or her. The same functionality may be extended to substantially any application or visual display.

Of course this could actually represent the next step for Flyover as in this patent a user is able to "walk around the display system and see different sides of the building" which isn't available today. On a table like display, like a PixelSense type of display table, this feature would be very cool.

For interest sake, one of the engineers behind this patent/invention is Ashley Saulsbury who was a distinguished engineer who worked at Sun Microsystems on streaming hours of uncompressed digital IMAX resolution movies onto 20' screens, to blasting high energy neutron beams at microprocessors.

Patent Credits

Apple credits Brett Bilbrey, Ashley Saulsbury (former Sun Microsystems/Oracle engineer) and David Simon as the inventors of patent application 20150138184 which was originally filed in Q4 2013. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Comments are reviewed daily from 5am to 7pm MST and sporadically over the weekend.