On November 15, 2012, the US Patent & Trademark Office published a patent application from Apple that reveals their ongoing work in advancing presence technology. Apple began their work on this project back in October 2009 and followed through with an extensive patent application in December 2011. In their latter application, Apple was exploring the use of Sonar technology and in their current work Apple reveals a number of other sensors being considered such as Radar. According to Apple, Radar is sensitive enough to detect a user's heartbeat in a room. Another kind of sensor listed could detect carbon dioxide emissions from a user's breath. The system will provide new and faster "awake" functionality, better power saving modes and even advance display interactions with a user. In one example, Apple states that as a user physically moves toward or away from their display, the imagery on their display will automatically zoom in or out. Being that this is Apple's third detailed presence related patent, we could safely say that it's becoming an important trend worth noting going forward.
Apple's Patent Background
Many computing devices are equipped with power saving features/modes intended to reduce power consumption when a user is not using the devices. Often, these power saving features are implemented though timers that count down a set amount of time from when the user last provides an input to the device. For example, a particular device may be configured to enter a sleep mode, or other mode that consumes less power than a fully operational mode, when a user has not provided input for five minutes.
Occasionally, however, a device may enter the power saving features/modes while a user is still using the device. For example, the power saving features may be entered because the user failed to provide input within the time period set for the timer while reading content on the device, viewing a movie, or listening to music. Additionally, recovery from the power saving feature/mode may take time, may even require the user to enter credentials, and generally may be a nuisance to the user.
Apple's Presence Sensing Solution
Generally, Apple's invention is directed to user presence determination and related computing device functionality. In some embodiments, power saving and/or power efficiency may be realized.
One embodiment may take the form of a computing device that is configured to sense the presence and/or absence of a user and provide an operating state based on the presence and/or absence of the user. In other embodiments, the computing device may calculate and provide a likelihood or probability score of the user being present or not present. In some embodiments multiple parameters may be determined, weighted, and used in conjunction in making a presence determination. This weighted detection can be used for more informed higher level decision making algorithms, or when fusing data from different sensors.
For example, in some embodiments, the computing device may be configured to determine when a user arrives or enters into proximity with the computing device and/or a probability that the user is present based on sensor input. With the probability calculation, false positives and/or false negatives may be reduced. Bayesian optimal thresholds may be implemented that achieve desired performance. In response to a positive determination that the user is present or upon achieving a threshold probability that the user is present, the device may power up, exit a sleep mode, and/or provide some feedback to the user.
In Apple's patent FIG. 1 noted above, is a computing device that may generally include one or more sensors that may be utilized for presence sensing. For example, one or more cameras and/or light sensors may be used in the presence sensing. Alternatively or in conjunction with the camera system, Apple may utilize other sensor types such as ultrasonic sensors, microwave RADAR, and so forth.
Sensor Able to Detect a Heartbeat in the Room
Some of these technologies may be utilized to determine physiological parameters when a human is in proximity to the device. For example, RADAR may be used to detect and/or locate a heartbeat in the room.
Depth Sensors
Apple's system may use various techniques and wavelengths of light in their system. For example, proximity may be determined using depth sensors. Some example depth sensor techniques may include: focusing and defocusing, active IR reflected power, active IR structured light; active IR time of flight (2D+depth), active IR time of flight (single pixel sensor), passive IR (motion detector), passive IR thermal imaging (2D), stereo vision, polarization techniques, and so forth.
In some embodiments, active IR may user multiple specific IR wavelengths to detect certain unique material properties, such as reflectivity of human skin or carbon dioxide emissions in breath. As such, the particular embodiments described herein are merely presented as examples and are not limiting.
A More Intelligent Awake System
Generally, one goal of Apple's presence sensing system may be to more intelligently use technology. For example, in some embodiments, a system awake may be initiated when it is determined that a user is approaching. The system awake may include a reduced set of routines so that the system is in an operational mode faster than with a conventional power up sequence. For example, the system may power up within a half second rather than six to eight seconds due to the reduced set of routines.
More Intelligent Power Saving Modes
In some embodiments, the computing device may be configured to determine when a user moves away from the device or leaves the proximity of the device. In response, the device may enter a power saving mode, such as a display sleep mode, a system sleep mode, activation of a screen saver, and so forth. Further, the system may exit the sleep mode partially in order to speed up the computer wake up time based on sensing the presence of a user.
Intelligently Tracking a User's Presence
In some embodiments, the device may also be configured to track the user movements (e.g., position and velocity) and, in response to certain movements, provide feedback and/or enter or change a state of operation. For example, movement toward the device may activate more features, such as providing more options/menus in a user interface, whereas movement away from the device may reduce the number of features available to a user, such as reducing the number of menus/options and/or reducing or increasing the size of the options displayed.
Intelligent Interactions with the Display based on Body Tracking & Environment
Additionally or alternatively, the display may zoom in or zoom out based on movement towards or away from the device. In some embodiments, a lateral movement of by the user (e.g., from left to right) may cause a change in a background and/or a screen saver image displayed on the device. Still further, the changing of the image may correspond generally with the sensed motion. For example, the movement from left to right may cause the image to be replaced in a left to right motion with another image. Alternatively, as a user moves from left to right, leaves or drapes may reflect the movement. That is, the leaves may be blown and tumble from left to right, or the drapes may sway in a manner corresponding to the detected movement.
Skin Tone Detection
Another embodiment may take the form of a method for determining if a user is in proximity of a computing device. The method includes capturing an image using an image sensor and computing at least one of the following from the captured image: a skin tone detection parameter, a face detection parameter, a body detection parameter and a movement detection parameter. The method also includes utilizing at least one of the skin tone detection parameter, face detection parameter and the movement detection parameter to make a determination as to whether a user is present and, if it is determined that a user is present, changing a state of the computing device.
Apple's patent FIG. 6 shown below is a flowchart illustrating a method for determining presence of a user; patent FIG. 7 is a flowchart illustrating skin tone detection routine for use in presence sensing. The detection method could also be used in low lighting situations.
Capturing 3D Images
In still another embodiment, a computing system is provided having a main processor and a presence sensor coupled to the main processor. The presence sensor includes an image sensor, and a processor coupled to the image sensor configured to process a captured image to determine if a user is present in the image. The image sensor may be configured to capture 3-D images, depth images, RGB images, grayscale images and/or so forth and one or more of the images may be used for a probability determination. Other inputs may also contribute information useful to the presence determination. For example, input devices such as a keyboard, mouse and/or microphone each may provide inputs useful to a presence probability determination. If the processor determines that a user is present in the image, an indication that a user has been determined to be present is sent from the processor to the main processor and the main processor changes a state of the computing system based on the indication.
Single Frame Motion Detection
Apple's patent FIG. 10 shown below illustrates frames being divided into window for single frame motion detection.
Multiple Concurrent Asynchronous Sensors
Apple's patent FIG. 12 is a flowchart illustrating an example method for multiple concurrent asynchronous sensors being used to determine a probability of a user being proximately located to a device.
Patent Credits
Apple's Sr. Director Mac Technology Software Myra Haggerty along with eight other engineers are noted as the inventors of this patent application which was originally filed under serial number 469996 in May 2012.
NOTICE: Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.
1. Samsung Invents Air-Gesturing Controls for Tablets & Beyond - 2. Samsung Invents Crazy Roly-Poly Mobile Docking Device - 3. Microsoft Envisions Air Gesturing Controlling Your TV & More and 4. Microsoft Creating Real-Time Biometrics for Gaming & Beyond
