Apple's Patent Background

Troxler fading is a phenomenon of visual perception that the human brain uses to cope with blind spots on the retina, which is the light-sensitive tissue lining of the inner surface of the eye. One result of this phenomenon is that when one fixates on a particular point, objects in one's peripheral vision will fade away and disappear. Another result related to this phenomenon is the perceived fading of a fixated stimulus when its retinal image is made stationary on the retina, which is otherwise known as a stabilized retinal image. This perceived fading causes the stimulus to fade away after a short time and effectively disappear to the viewer (i.e., the person whose retina is capturing and perceiving the fixed stimulus).

The perceptual fading of stabilized retinal images presents a particularly difficult problem with respect to the navigation of a GUI using a movable indicator, such as a cursor, whose position is controlled by eye tracking.

When eye tracking is accurate such that the cursor consistently appears wherever the user looks in the GUI, the cursor will be stationary with respect to the user's retina, and as such the cursor may fade with respect to the user's perception of the GUI. The perceptual fading of the cursor may cause the user to lose track of the location of the cursor within the GUI, which lessens the usability of the GUI itself.

When a user loses track of a cursor within a GUI, systems typically depend on the user to provide manual input to the GUI in order to remind the user of the location of the cursor. However, this is undesirable as it causes the user to become distracted from whatever interaction they are trying to accomplish with the GUI in the first place.

Further, such systems don't take into account when the perceptual fading of the cursor occurs, and how it can be prevented such that the user does not notice or consider the prevention measure to be a nuisance (e.g., by blocking other information within the GUI or becoming a distraction).

Apple Advances Eye Tracking System

Apple's invention in various implementations provides systems, methods, and devices that provide a user of a GUI with one or more measures to counteract a perceptual fading of a cursor with respect to the GUI.

The GUI may exist as part of an interactive operating system environment such as Mac OS X. The cursor or movable indicator may be any object that the user manipulates in order to navigate or interact with the GUI, such as a mouse cursor, bi-directional text cursor, a movable object within a game, or any other object within the GUI.

In certain configurations, the GUI may be presented by and controlled with an eye tracking system. The eye tracking system may allow a user of the GUI to navigate or interact with various elements in the GUI, such as a word processor, game, web browser, or any other suitable interactive application, simply by gazing or looking at a particular point on a display of the GUI.

For example, the user may gaze at a button on the menu bar of a word processing application, causing the eye tracking system to render a cursor over the button. In certain configurations, the user of the eye tracking system may be able to select the button using any suitable input device external to the display device that is presenting the GUI, such as a track pad or mouse. In other configurations, the user of the eye tracking system may be able to select the button using an input device built-in to the display device presenting the GUI itself, such as a capacitive or resistive touch screen. In yet other configurations, the user of the eye tracking system may be able to select the button using facial gestures or voice input.

In certain configurations, the eye tracking system may persistently render the cursor wherever the user looks in the GUI. This rendering of the cursor may be accurate to the degree that the cursor becomes a stabilized retinal image with respect to the user's eyes. As such, the cursor may fade with respect to the user's perception of the GUI. In other words, the cursor may no longer be visible to the user. In such situations, it is desirable to restore the user's perception of the cursor to counteract this fading effect. Accordingly, the eye tracking system described in Apple's patent may automatically alter the position, appearance, or both of the movable indicators so that it is no longer a stabilized retinal image and can be perceived by the user.

Apple's patent FIG. 1A noted below is a diagram including components of an eye tracking system 100A. In Apple's patent FIG. 1B we're able to see a diagram including components associated with a display device or eye tracking apparatus 110B of the eye tracking system.

In certain configurations, Apple states that the eye tracking apparatus, noted as #106 of FIG. 1A, may include hardware, such as cameras or infrared transmitters and receivers, which are integrated into the display device itself (#104).

In other configurations, eye tracking apparatus may include hardware that is free standing from display device, such as a headset (see headset) or web camera.

The eye tracking system may apply to a desktop computer, MacBook, iPad, iPhone, iPod touch or any suitable computing form factor (headset, television, wearable computer).

Apple notes that the display device or eye tracking apparatus noted in patent figure 1B-100B may be in communication with data storage (#120) through a data link (#119). The IR transmitter (#112) may include hardware, software, or a combination therefore arranged to generate and transmit infrared or near-infrared signals from display device or eye tracking apparatus towards an object of interest such as a user's face, or more particularly, a user's eyes.

In other configurations, reflections of these infrared or near-infrared signals may be used to detect the presence or absence of a feature of a user's eyes, such as the presence or absence of an exposed pupil.

The Key Countermeasure Generator

At the top of our report we presented Apple's patent background that pointed to "Troxler fading" as a visual perception phenomenon that when one fixates on a particular point, objects in one's peripheral vision will fade away and disappear. Apple's eye tracking system includes a key countermeasure generator which is seen in patent FIG. 1C under #138 which is designed to counter the Troxler fading phenomenon.

Basically the countermeasure generator receives infrared and camera data as input, determines whether a user's perception of a cursor within a GUI presented by the eye tracking system is subject to fading, and communicates data to the input device driver or GUI rendering module in order to provide a countermeasure (i.e., a counteracting measure) to reduce, prevent, or stop Troxler fading.

In certain configurations, the countermeasure generator may be configured to analyze infrared and camera data to recognize facial features associated with a user such as, for example, the presence or absence of an exposed pupil, a pigment in the eye, or an eyelid. The countermeasure generator may then track the facial features over time such that a determination can be made as to whether the user closed and subsequently opened their eyes, signifying a blink. The countermeasure generator may then prompt input device driver or GUI rendering module to alter the position of the cursor in order to counteract a perceptual fading of the cursor based on detecting this blink.

Apple's patent FIG. 1D noted above is a diagram 100D of a more detailed view of the countermeasure generator within the computer processing environment 100C of FIG. 1C.

According to Apple, the blink tracking module, noted as # 142 of patent FIG. 1D, may employ one or more computer vision techniques, such as image filtering, edge detection, thresholding, object recognition, or color processing in order to detect the presence or absence of a user's exposed pupils, eyelids, or a pigment of the user's eye in order to detect whether the user's eyes are open and subsequently close. In certain configurations, the blink tracking module may make this determination using infrared images of a user's eyes.

Eye Tacking System for the Car, Video Game Console or Entertainment System

Apple's patent FIG. 2A note below is a diagram of a side view of a user interacting with a GUI presented by Apple's eye tracking system.

Interestingly, beyond devices such as an iPhone, iPad or MacBook Air, Apple specifically adds that the display and imaging components #208 noted in patent FIG. 2B may be integrated within the packaging of other devices or structures such as a vehicle, video game system and entertainment system.

Apple's eye tracking system could complement another of Apple's entertainment or gaming system inventions such as this wild 3D imaging system.

Apple notes that in certain configurations, the user will be able to control the position of a cursor within the GUI presented by the eye tracking system on a display using their eyes. In such configurations, imaging components (#208) may track the movements of the user's eyes (#210) and determine the user's point of gaze using any suitable eye tracking technique, including but not limited to Pupil Centre Corneal Reflection (PCCR) using dark pupil eye tracking, bright pupil eye tracking, or both.

In certain configurations, when the user's point of gaze transitions from a first point of gaze to second point, the cursor glides between the positions in the GUI associated with these points of gaze.

Apple's patent FIG. 2B is a diagram of an overhead view of a user interacting with a GUI presented by the eye tracking system. In Apple's patent FIGS. 3A and 3B, the "movable indicator" is rendered by the GUI in the form of cursor. Apple's patent FIG. 6 is one of their flowcharts of a process for counteracting a perceptual fading of a movable indicator within a GUI presented by the eye tracking system.

Patent Credits

Apple credits David Julian as the sole inventor of this patent application which was originally filed in Q2 2012. For review Apple's 36 patent claims and full details, review patent application 20130265227.

Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.

New on Patent Bolt this Week

Microsoft was hammered by Six Patent Troll Lawsuits this Week

Cellular Communications Equipment Sues Microsoft with Nokia Patent

Samsung Granted New Design Patents for e-Pen, Tablet & More

Google Seeks Patent for Specialized In-Vehicle Gesture System