Microsoft's Patent Background

User enjoyment of video games and related media experiences can be increased by making the gaming experience more realistic. Previous attempts to make the experience more realistic have included switching from two-dimensional to three-dimensional animation techniques, increasing the resolution of game graphics, producing improved sound effects, and creating more natural game controllers.

So where will Microsoft take gaming in the future? According to a recent patent application filing, Microsoft envisions a totally immersive experience: Specifically, the patent states that "An immersive display environment is provided to a human user by projecting a peripheral image onto environmental surfaces around the user. The peripheral images serve as an extension to a primary image displayed on a primary display."

Microsoft's Future Immersive Gaming System

Microsoft's patent FIG. 1 shown below schematically shows an embodiment of a display environment (100). The display environment is depicted as a room configured for leisure and social activities in a user's home. In the example shown in FIG. 1, the display environment includes furniture and walls, though it will be understood that various decorative elements and architectural fixtures not shown in FIG. 1 may also be present.

As shown in Microsoft's patent FIG. 1, a user (102) is playing a video game using an interactive computing system (110) (such as the Xbox gaming console) that outputs a primary image to the user's HDTV (104) and projects a peripheral image on environmental surfaces (e.g., walls, furniture, etc.) within the display environment via environmental display 116.

In the gaming scenario shown in FIG. 1, the user is playing a first person shooter video game and is focused on the primary images of the game that are shown on their TV. Connected to a future version of Microsoft's Xbox is the environmental display which is shown to be sitting on top of the television. It will be configured to display a peripheral image on environmental surfaces.

The peripheral image appearing on the user's wall will be configured to appear to be as an extension of the primary image that is shown on their TV. As a user perceives the peripheral image with the user's peripheral vision, the user may be situationally aware of images and objects in the peripheral vision while being focused on the primary image.

In the example shown in patent FIG. 1, we see that the user is focused on the gameplay on his TV but may be aware of an approaching video game enemy from the user's perception of the peripheral image displayed on his back wall. In some embodiments, the peripheral image is configured so that, to a user, the peripheral image appears to surround the user when projected by the environmental display. Thus, in the context of the gaming scenario shown in FIG. 1, the user may turn around and observe an enemy sneaking up from behind.

In the embodiment shown in FIG. 1, the environmental display is a wide-angle RGB projection display device configured to project a peripheral image in a 360-degree field around the environmental display.

Immersive 3D Environments

Microsoft states that while the example primary display and environmental display shown in FIG. 1 includes 2-D display devices, "it will be appreciated that suitable 3-D displays may be used without departing from the scope of the invention." Furthermore, Microsoft states that in some embodiments, "the user may enjoy an immersive 3-D experience using suitable headgear, such as active shutter glasses configured to operate in synchronization with suitable alternate-frame image sequencing at primary display and environmental display."

The patent filing goes on to state that in some embodiments, immersive 3-D experiences may be provided with suitable complementary color glasses used to view suitable stereographic images displayed by the primary and environmental displays.

In some embodiments, the user may enjoy an immersive 3-D display experience without using headgear. For example, the primary TV display may be equipped with suitable parallax barriers or lenticular lenses to provide an autostereoscopic display while the environmental display renders parallax views of the peripheral image in suitably quick succession to accomplish a 3-D display of the peripheral image via "wiggle" stereoscopy.

The System's Depth Camera

Microsoft notes that a future version of their Xbox will be able to connect the environmental display which incorporates a depth camera (114) as we note below in our closeup graphic. The depth camera is configured to generate three-dimensional depth information for the total display environment such as the user's four walls.

According to Microsoft, the depth camera may be configured as a time-of-flight camera configured to determine spatial distance information by calculating the difference between launch and capture times for emitted and reflected light pulses. Alternatively, in some embodiments, Microsoft states that the depth camera may include a three-dimensional scanner configured to collect reflected structured light, such as light patterns emitted by a MEMS laser or infrared light patterns projected by an LCD, LCOS, or DLP projector.

In some embodiments, the depth camera may include a plurality of suitable cameras to capture three-dimensional depth information within the display environment. In some embodiments, the depth camera may include image processing software configured to stitch a panoramic image from a plurality of captured images.

In the embodiment shown in FIG. 1, the depth camera shares a common housing with the environmental display (116). By sharing a common housing, the depth camera and the environmental display may have a near-common perspective, which may enhance distortion-correction in the peripheral image relative to conditions where the depth camera and the environmental display are located farther apart. However, it will be appreciated that the depth camera may be a standalone peripheral device operatively coupled with a future version of their Xbox system.

Wild Kinect Connection: The Tracking System

As shown in our close-up graphic above, the Xbox is operatively connected with a user tracking device 118, which is really Microsoft's Kinect. Kinect may include a suitable depth camera configured to track user movements and features (e.g., head tracking, eye tracking, body tracking, etc.). In turn, the Xbox may identify and track a user position and act in response to user movements detected by Kinect. Thus, gestures performed by the gamer while playing a video game may be recognized and interpreted as game controls. In other words, Microsoft's Kinect will allow the gamer to control the game without the use of conventional, hand-held game controllers.

In some embodiments where a 3-D image is presented to a user, Microsoft's Kinect may track a user's eyes to determine a direction of the gamer's gaze. For example, a gamer's eyes may be tracked to comparatively improve the appearance of an image displayed by an autostereoscopic display within the gamer's television or to comparatively enlarge the size of a stereoscopic "sweet spot" of an autostereoscopic display within the television relative to approaches where a gamer's eyes are not tracked.

It will be appreciated that, in some embodiments that Microsoft's Kinect may share a common housing with environmental display (116) and/or depth camera (114).

Mapping the Gamer's Environment

Microsoft's patent FIG. 2 presents an overview of the method 200 which may comprise displaying a distortion-corrected peripheral image (206). In such embodiments, the display of the peripheral image may be adjusted to compensate for the topography and/or color of environmental surfaces within the display environment.

In some of such embodiments, topographical and/or color compensation may be based on a depth map for the display environment used for correcting topographical and geometric distortions in the peripheral image and/or by building a color map for the display environment used for correcting color distortions in the peripheral image. For example, structured light may be projected on the walls, furniture and both the decorative and architectural elements found in the gamer's entertainment room.

Shielding the Gamer's Position in the Environment

As noted in the Method flowchart above, displaying the peripheral image by the new environmental display system may include shielding a portion of the gamer's position from light projected by the environmental display. In other words, projection of the peripheral image may be actually and/or virtually masked so that a gamer will perceive relatively less light shining from the peripheral display to the gamer position. This may protect the gamer's eyesight and may avoid distracting the user when moving portions of the peripheral image appear to be moving along the user's body.

In some of such embodiments, Microsoft's Kinect tracks a user position using the depth input received from the depth camera and outputs the peripheral image so that a portion of the gamer position is shielded from peripheral image light projected from the environmental display. Thus, shielding a portion of the gamer position may include determining the gamer position. The user position and/or outline may be identified by the gamer's motion relative to the environmental surfaces of the display environment, or by any suitable detection method. The gamer position may be tracked over time so that the portion of the peripheral image that is shielded tracks changes in the gamer position.

In Microsoft's patent FIGS. 4 and 5 noted above, we see a schematic of a display environment in which a peripheral image 302 is being projected at time T.sub.0 (FIG. 4) and at a later time T.sub.1 (FIG. 5). For illustrative purposes, the outline of gamer 102 is shown in both figures with gamer 102 moving from left to right as time progresses.

A shielded region shown in outline for illustrative purposes only, tracks the gamer's head, so that projection light is not directed into the gamer's eyes. While patent FIGS. 4 and 5 depict a shielded region as a roughly elliptical region, it will be appreciated that the shielded region (602) may have any suitable shape and size.

Microsoft's patent application was originally filed in Q1 2011 and published by the US Patent and Trademark Office in Q3 2012. 

What are Your Thoughts on This?

I could remember it being so dramatic when gaming shifted from the CRT to HDTV with 1080p. Games came to life like never before. So the very thought that gaming will one day be able to immerse us into detailed gaming environments by projecting the imagery of the game onto the walls of our room is simply wild. Just having it projected unto the wall behind our HDTV's would be a great start. But having it truly immerse us with the gaming enviroment taking over all four walls around us could be a real breakthrough in the gaming experience. For now it's a theory, but the evolution of gaming says it's possible. What are your thoughts on this?     

Note to Referring Sites: We ask that referring sites limit the use of our graphics to a single graphic. Thank you for your cooperation. 

Notice

Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.