Today the US Patent & Trademark Office published a patent application from Apple that reveals the next chapter for Apple's OS X interface beyond Snow Leopard. Apple's patent introduces us to a higher end 3D interface that will be able to provide parallax transforms of objects, applications and even games with incredibly high detailed reflection realism using a new wide-angle iSight camera. The camera tracks the users head and changes the screen perspective accordingly. In our October report titled Intel, Apple working on 3D Internet we discussed the coming 3D internet in context with Intel's coming 32 nm Microarchitecture code named Sandybridge. Intel has made it clear that it will take 8 core systems to bring about the 3D internet experience for consumers. Likewise, Apple's patent makes several references to the enormous power it will take to bring about this next generation interface experience that will bring in new realism to 3D that will also enhance our Augmented Reality experiences. Apple's patent also introduces us to all new Sensing and Sound detecting mechanisms to enhance our 3D experience. This is one powerful patent that shows us what we could expect from OS X in the coming years.
New 3D Interface System
Apple's patent FIG. 1 is a schematic view of an electronic device that may include a display, a sensing mechanism (104) and control circuitry (110). It may also include communications circuitry and/or other components.
Apple's patent states that device 100 may work with any suitable device operative to display information to a user. The display may be associated with an iPod, iPhone, any MacBook, iMac/Mac Pro desktops and so forth.
Device 100's display may include any suitable screen or projection system for providing a display visible to the user. That could translate into an LCD, a movable display or a projecting system for providing a display of content on a surface remote from the electronic device. A June 2009 patent application of Apple's first presented us with information regarding a pico projector and in yesterday's report on Augmented Reality, a video presentation by MIT's Pranav Mistry illustrated how a pico projector projected photos from a SixthSense device onto an adjacent wall so that he could manipulate the photos just taken from his SixthSense-camera and then email them to a colleague or friend from the projected interface on the wall. In today's patent, Apple's ability to do such will be enhanced using a unique 3D environment that could be projected onto a wall or just seen on a computer's display.
The Sensing Mechanism
The Sensing mechanism shown above as patent point 106 may include any suitable mechanism for detecting and sensing the user. For example, the sensing mechanism may be operative to detect the position of the user's face relative to the display. As another example, the sensing mechanism may be operative to detect a particular feature of the user's face (e.g., the location of a user's eyes or mouth, whether the eyes or mouth are open, or what the eyes are looking at). As still another example, the electronic device may detect the position of a user's hands, arms, or other body part. As still yet another example, the electronic device may detect or sense the position of the user (e.g., the distance of the user from display 104, or the position of the user relative a portion or element of display 104).
Suitable sensing mechanisms may include, for example, an optical sensing mechanism (e.g., a camera operative to detect light), a mechanism operative to detect non-visible wavelengths (e.g., IR and UV radiation), a pressure sensing mechanism, a sound detecting mechanism (e.g., one or more microphones operative to detect from where sound is coming), any other suitable sensor for detecting a physical aspect of the user or the user's surroundings, or any other suitable mechanism.
New Wide-Angle iSight Camera
In some embodiments, the sensing mechanism may be incorporated in or [on an] adjacent display 104 (e.g., a camera incorporated above a laptop screen), another component of electronic device 100, or near any other portion of the electronic device. In one implementation, sensing mechanism 106 may include a wide-angle lens embedded above display 104 (e.g., such that the user viewing the display may be framed by the lens).
The use of a camera is the corner stone of presenting Augmented Reality. In yesterday's report on this subject, as noted above, a Coke Zero Avatar AR video was presented that makes this point crystal clear. The camera is able to sense the user's positioning so that the animated aircraft is in proportion to the screen and is also able to be manipulated by the user in such things as directing the aircraft's movements.
In respect to a camera's use, the patent states that using a camera - the electronic device may detect the user's face from the background (e.g., based on particular attributes of a human face, or based on the difference in shape or color between the background and the user), and determine the position of the user's head in the field of view of the camera. As another example, the electronic device may use an IR or heat-sensitive camera, or an optical camera with an adequate lens to distinguish the user's body (e.g., the user's head or torso) from the background to determine the position of the user relative to the camera field of view. This approach may be particularly useful in low light or night situations.
The Control Circuitry
The Control circuitry may be operative to control the operations and functions of electronic device 100. The Control circuitry may include, for example, a processor, a bus (e.g., for sending instructions to the other components of electronic device 100), memory, storage, or any other suitable component for controlling the operations of the electronic device. In some embodiments, a processor may drive the display and process inputs received from the user interface or input mechanism.
Control circuitry 110 may be operative to execute instructions provided by firmware, software, or both implemented in the electronic device. For example, the control circuitry may be operative to execute instructions provided by a communications application (e.g., an email or telephone application, or a remote access application), word processing application, a media playback application (e.g., a music, radio or video playback application), a system utility, a location-detection application, or any other suitable application for controlling operations of the electronic device. As another example, the control circuitry may be operative to manage and direct the display of on-screen elements based on outputs of sensing mechanism 106.
In some embodiments, the input mechanism may include a mechanism that is remotely coupled to the display or communications circuitry. For example, the input mechanism may include a keyboard, keypad, mouse, remote controller, voice-instruction apparatus, or any other mechanism for providing inputs.
3D Perspective: The Parallax Approach
As the user moves relative to the electronic device, the sensing mechanism may detect the user's movement and change the display based on the movements. For example, Apple's patent FIG. 2 shown below is that of a schematic view of an illustrative display when the user is aligned with the center of the display. Patent FIG. 3 is a schematic view of a display to which a parallax transform is applied when the user moves to the right. Patent FIG. 4 is a schematic view of a display to which a parallax transform is applied when the user moves to the left.
The parallax approach illustrated in FIGS. 3 and 4 below may be applied to change the appearance of icons displayed on the display based on the detected position or movement of the user. For example, Apple's OS X dock will be able to present the user with a true 3D perspective no matter where the position of the user is to the display. Wikipedia presents an animation that clarifies this technology approach.
In particular, the electronic device may use three-dimensional graphic engines (e.g., applying the OpenGL specification) to perform perspective projections of three-dimensional objects displayed on a two-dimensional display (e.g., display 1300 or 1400). In some embodiments, the electronic device may apply a three-dimensional transform matrix to elements of displays 1300 and 1400 based on the determined position of the user. For example, the electronic device may detect the current position of the user using the sensing mechanism and define, from the detected position, a location and rotation (e.g., using 3D vectors) from which to calculate the transform. The electronic device may then use the defined location and rotation vectors to transform each relevant point of a displayed object and create a perspective transform of the object.
Take note of each of the graphics lower left corner. What you're seeing is how the new iSight camera will target your head and change the perspective of the screen accordingly. This is very, very sophisticated and cool technology. (Note that you could click on this graphic once, and when you see the image, click on it again to get a larger image. No, you can't double click the image to see that.)
Realism: Reflecting Light on 3D Surfaces
In some embodiments, the electronic device may first determine the user's position (e.g., from the output of the sensing mechanism) relative to the screen to determine the angle and location from which the user should appear to see light impacting the displayed object. For example, the electronic device may define one or more sources of directed light (e.g., a light source opposite the center of the display), or may instead or in addition define a diffuse or generalized light source for the object. Once the electronic device has determined the position of the light source and the position of the user, the electronic device may map the light to each surface of the electronic device, and determine from the physical properties of each surface, as well as the shape and orientation of each surface, the manner in which light will reflect from each surface (e.g., using Fresnel equations). Because the processing power and time required to trace each ray of light as it moves through or off the displayed object and reaches the user's eye may be too large for a consumer electronic device, the electronic device may use any suitable approximation to allow the electronic device to change the display quickly while providing a reasonably accurate representation of the light's reflection. For example, display 1900 may include reflection 1920 depicting light reflecting from object 1910 in the orientation of display 1900, while display 2000 may include reflection 2020 depicting the same light reflecting from a different portion of object 1910 (e.g., because object 1910 was rotated).
In addition, the electronic device may determine, from the physical properties and shape of each surface of the object, the manner in which image 1932 may be reflected on different surfaces of object 1910. For example, surface 1912 may be a polished metal surface that is highly reflective, and may reflect image 1932 well. As shown in display 1900, surfaces 1912b, which may be curved, may include reflections 1936 of user 1934. Surface 1912a - which is substantially straight - may include a different reflection 1937 of user 1934. Each reflection 1936 and 1937 may be distorted, for example based on the shape of each surface from which the image of user 1934 is being reflected. Other surfaces, for example surface 1913, may be diffuse such that light may be reflected, but no discernable part of image 1932 (e.g., as shown in FIG. 20, where the user's position relative display 2000 and the orientation and shape of surface 1912 is such that the detected image is not reflected back to the user).
The electronic device may enable transformations for changing the user's perspective using any suitable approach. In some embodiments, transformations may be performed only when the sensing mechanism is enabled. For example, if a camera is not enabled, or if a camera is not able to detect the user (e.g., because the user is out of the frame of view, or because the background and user are too similar), the electronic device may not perform transformations. In some embodiments, transformations may only be available when three-dimensional objects are displayed. For example, transformations may only be enabled when the user navigates to an application screen (e.g., a full screen view of a three-dimensional icon, or a request to display a dock including at least one three-dimensional icon).
Realism: Gaming Applications
In some embodiments, transformations may be enabled based on the application or objects displayed. For example, particular applications, such as video games (e.g., a car racing game or other game in which the user plays in the first person) may be associated with particular transformations and enable particular transformations based on the current mode of the application. In one approach, a user of a car racing or flight simulator video game may be able to look out different windows, or see different perspectives of the track or sky based on the user's position relative a sensing mechanism. In another approach, a user viewing a three-dimensional object (e.g., using a web-based shopping application) may view different perspectives of the object as the user moves relative to the display screen. In still another approach, a user creating a three-dimensional object using a design program may view different perspectives of the created object by moving relative to the sensing mechanism.
An Example of Head Tracking on an iPhone/iPod Touch
Notice: Patently Apple presents only a brief summary of patents with associated graphic(s) for journalistic news purposes as each such patent application and/or grant is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application and/or grant should be read in its entirety for further details. For additional information on any patent reviewed here today, simply feed the individual patent number noted above into this search engine. To read this report in another major language, use Yahoo! Babel Fish.
Please note that any of the temporary links presented in today's report are in fact temporary and may redirect you to unrelated patents in the future. In such cases refer back to the search engine instructions above.
Comments