On December 29, 2011, the US Patent & Trademark Office published a patent application from Apple that reveals one of the next chapters for device security. In 2009, Apple's presence detection patent first came to light in relation to future MacBooks. Then in November of this year, Apple revealed a heavy duty 3D face and object recognition system that could be used for home and enterprise security applications. In today's revelations, Apple introduces us to a more down to earth and practical security system for our portable devices. For simple home or personal use, the system could be setup to recognize your presence and face to quickly turn on your device. This would bypass the need for entering a password or even having to touch the home button to get to your homepage. For use at work, the facial recognition system could be set to higher levels of security. All in all it sounds like a very promising security system is in our future.
The Problems of Face Recognition Apple Seeks to Solve
Most face recognition systems fall into one of two categories. A first category system tends to be robust and could tackle various lighting conditions, orientations, scale and the like, and tends to be computationally expensive. A second category system is specialized for security-type applications and could work under controlled lighting conditions.
Adopting the first category systems for face recognition on consumer operated portable appliances that are equipped with a camera would unnecessarily use an appliance's computing resources and drain its power. Moreover, as the consumer portable appliances tend to be used both indoor and outdoor, the second category systems for face recognition may be ineffective. Such ineffectiveness may be further exacerbated by the proximity of the user to the camera, i.e., small changes in distance to and tilt of the appliance's camera dramatically distort features, making traditional biometrics used in security-type face recognition ineffective.
A Basic Overview of Apple's Solution
One aspect of Apple's invention could be implemented in methods performed by an image processor that include the actions of processing a captured image of a face of a user seeking to access a resource by conforming a subset of the captured face image to a reference model. The reference model corresponds to a high information portion of human faces. The methods further include comparing the processed captured image to at least one target profile corresponding to a user associated with the resource, and selectively recognizing the user seeking access to the resource based on a result of said comparing.
These and other implementations could include one or more of the following features. In some cases, the high information portion includes eyes and a mouth. In some other cases, the high information portion further includes a tip of a nose. Processing the captured image could include detecting a face within the captured image by identifying the eyes in an upper one third of the captured image and the mouth in the lower third of the captured image.
The reference model includes a reference image of a face, and processing the captured image further could include matching the eyes of the detected face with eyes of the face in the reference image to obtain a normalized image of the detected face. Additionally, processing the captured image could further include vertically scaling a distance between an eyes-line and the mouth of the detected face to equal a corresponding distance for the face in the reference image in order to obtain the normalized image of the detected face. In addition, processing the captured image could further include matching the mouth of the detected face to the mouth of the face in the reference image in order to obtain the normalized image of the detected face.
In some implementations, comparing the processed captured image could include obtaining a difference image of the detected face by subtracting the normalized image of the detected face from a normalized image of a target face associated with a target profile. Comparing could further include calculating scores of respective pixels of the difference image based on a weight defined according to proximity of the respective pixels to high information portions of the human faces. The weight decreases with a distance from the high information portions of the human faces. For example, the weight decreases continuously with the distance from the high information portions of the human faces. As another example, the weight decreases discretely with the distance from the high information portions of the human faces. As yet another example, the weight decreases from a maximum weight value at a mouth-level to a minimum value at an eyes-line.
In some implementations, processing the captured image could include applying an orange-distance filter to the captured image, and segmenting a skin-tone orange portion of the orange-distance filtered image to represent a likely presence of a face in front of the image capture device. Processing the captured image could further include determining changes in area and in location of the skin-tone orange portion of the captured image relative to a previously captured image to represent likely movement of the face in front of the image capture device. Also, processing the captured image further could include detecting a face within the skin-tone orange portion of the orange-distance filtered image when the determined changes are less than predetermined respective variations.
The Advantages of Apple's Invention
Particular implementations of the subject matter described in this specification could be configured to realize one or more of the following potential advantages. The techniques and systems disclosed in this specification could reduce the impact of lighting and emphasize skin variance. By acquiring images with the appliance's own image capture device, the approximate location and orientation of face features could be pre-assumed and could avoid the overhead of other face recognition systems. The disclosed methods could ignore face biometrics, and rather use feature locations to normalize an image of a test face. Further, the face recognition techniques are based on a simple, weighted difference map, rather than traditional (and computationally expensive) correlation matching.
The iOS Device Acknowledging Presence
Apple's patent FIG. 1 shown below illustrates an iPhone with its face side camera sitting in a dock. The iPhone could be turned off while the camera could remain on. Panel 100 shows a potential user who is approaching the iOS device sitting in its dock. In response to the iPhone detecting (120) that the potential user has stopped in front of its face side camera, the iPhone will be able to transition to a new state (shown in patent point 102') to acknowledge the presence and attention of the potential user.
In some implementations, the iOS device acknowledges the presence of the potential user by turning on the display. Further in response to detecting the presence of the potential user, the iOS device could trigger a subsequent process for recognizing the potential user's face.
Could be configured to Recognize Faces in your Family or Workplace
Furthermore, the iOS device could be configured to recognize faces of a predetermined group (including a small number) of users that may login on the iOS device and could present each user with a personalized configuration (see patent point 142). For example, to comply with such personalized configurations, the iOS device could modify screen saver slide shows or other appliance non-security preferences.
I also see that patent point 142 noted above shows us that we could be greeted with a message saying Hello! Although that will likely be optional, I'm curious as to how Apple will implement such a feature. Will they use a text only greeting or will they venture out and introduce a more personalized greeting from Siri (if you're an iCloud subscriber, that is). Depending on your view of talking devices, that could either be cool or creepy (ha!).
Apple's Method taps into the GPU of an iOS Device or MacBook
The methods disclosed in Apple's patent specification could adequately recognize a user associated with an iOS device without computing resources overhead that is characteristic of other face recognition techniques. Therefore, the face detection and recognition methods described in Apple's specification could be implemented in hardware, for example in graphical processing units (GPUs) of the iOS device. Apple clarifies that the new face detection and recognition system will apply to the iPhone, iPad, iPod touch and MacBook.
Apple's patent FIG. 4 shown below illustrates a flowchart for detecting potential presence of a person's face in front of an image capture device and for inferring the person's level of attentiveness. The latter feature is likely to secure against those trying to access an iOS device with a simple picture, as was the case with Google's failed Android feature.
2D Correlation Matching System
Apple's patent FIG. 7 shows an example of a process for matching a normalized image of a test face with a normalized image of a target face.
The normalized image (see patent point 644) of a test face could be iteratively compared with each of target faces (see patent point 710) based on two-dimensional correlation matching (patent point 750). The 2D-correlation matching used in this method is based on generating a difference map between the two images, in contrast to other complex correlation matching algorithms that are based on correlations of the underlying face-shapes. Moreover, the 2D-correlation matching could be performed in accordance with weighting rules and scoring rules maintained by the computerized device configured to execute the process (see patent points 730 and 740).
And lastly, Apple's new security features could, in logical theory, do away with the necessity to use the classic iOS device feature known as "Slide to Unlock." Of course it will depend on the speed of the facial recognition system. If it's fast enough, I know that I'd definitely prefer that option over the process of hitting the home button and then sliding the bar over to unlock my idevice to get to the home screen. Only in time will we know if that pans out, but it sure sounds very promising indeed.
Apple's patent applicationwas originally filed in Q2 2010 by inventor Robert Mikio Free.
Notice: Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. Apple's patent applications have provided the Mac community with a clear heads-up on some of Apple's greatest product trends including the iPod, iPhone, iPad, iOS cameras, LED displays, iCloud services for iTunes and more. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.
The Great Community Sites covering our Original Report
MacSurfer, Twitter, Facebook, Apple Investor News, Mac Hash, Planet Geek, MacPrime Switzerland, iPadworld, ApfelBlog Switzerland, Phandroid, Google Reader/news Google, Ameblo Japan, Macnews, iPhone World Canada, MarketWatch, Aberto ate de Madrugada Portugal, Zive Czech Republic, DSL Reports, Real Clear Technology, The New York Times Around the Web section, New York Times BlogRunner, MacDailyNews, 9to5 Mac, Ubergizmo, MacRumors, GigaOM, BGR, Business Insider, iClarified, everythingiCafe, SlashGear, TUAW, ReadWriteWeb, Pocketnow, Pulse2, and iSpazio Italy.
Techmeme, VentureBeat, iDownloadBlog, SecurLinx and The Mac Observer. Saggiamente Italy, iFun Germany, Textually.org, Onliner Russia, Gopego Indonesia, Hardware.Info Dutch, Test Mobile France, Appbuntu Indonesia, Wired - Gadget Lab, Businessweek,, iGeneration France, ETNews Korea, iPhoneros Spanish, iPhone Hacks, TorrentRG Korea, ApfelTech Germany, One More Thing Netherlands, Esmandau Spanish, Recombu, iPhone Buzz, Fierita Links Spanish, Cult of Mac, Chip Online Germany, Rossiyskaya Gazeta Russia and Engadget Spain.
Spiegel Online Germany, iPhone Rumors Italy, New MobileLife China, Olhar Digital Portuguese, Macnews Germany, CydiaHelp, and Movil Zona Spain, Appllio Japan, iPhoneRoot Russia, Mac1 Norway, UltraSnow Canada, 5 TeleCino Spain, Consomac France, e24 Norway, Tom's Guide, iPhoneWorld Spain, IT.TUT Belarus, GottaBeMobile, The Mobile Indian India, GeekyGadgets, Gizmodo Germany, DigiSecrets, WinFuture Germany, AppAdvice, OnlyGizmos, NewsPress Montenegro (English), Portaltic Spain, PFSK, D|Zone Netherlands, Zone Numerique France, Wired UK, and more.
Note: The sites that we link to above offer you an avenue to make your comments about this report in other languages. These great community sites also provide our guests with varying takes on Apple's latest invention. Whether they're pro or con, you may find them to be interesting, fun or feisty. If you have the time, join in!
