Today the US Patent & Trademark Office published a patent application from Apple that relates to a completely new Home-based invention that primarily covers a new outdoor camera system that communicates to those in a home who is at the front door via a HomePod mini, Smart TV, Apple TV, iPhone, iPad and more. Apple's inventors are mostly from their machine learning and computer vision teams.

Apple's patent application covers techniques for providing or suppressing a notification when a particular person is detected at a particular location. For example, a user may want to be notified if someone they recognize is at the front door.

However, in other instances, they may not want to bothered if the person at the door is someone they recognize. In one example, video or still images of a person may be captured by a computing device (e.g., at least including a camera). In some examples, if that person is detected as a particular known person, the computing device may determine to notify the user about the particular person being at the particular location – the front door for instance. That way, the user will know that someone they know/recognize is at the front door.

However, in other examples, the computing device may determine to suppress the notification if the detected person is recognized, thus avoiding unnecessary notifications (e.g., the user may not care to be notified if their spouse or roommate is detected at the front door).

A second part of the patent covers a device like a HomePod mini being able to capture images. In some examples, a resident device (e.g., a home automation device such as a smart speaker, a digital media player, or other appliance) may receive one or more image croppings from a user device (e.g., a mobile phone). The image croppings may be generated from a plurality of images that are managed as part of a library of images (e.g., a photo library) stored on the user device, and including contacts associated with the user device.

Each image cropping may have been selected for reception by the resident device based at least in part on a determined level of information gain associated with the particular image cropping, which may enable the resident device to perform facial recognition of the face of the first person.

The resident device (HomePod mini) may also receive one or more images from another device that includes a second camera (e.g., a home observation camera), whereby the second camera has a viewing area including a particular location (e.g., a home's front door porch).

The one or more images may, respectively, include a portion of a face of a person at the front door, whose identity is not yet determined. The resident device (HomePod mini) may then determine a score that corresponds to a level of similarity between a first set of characteristics associated with the face of the first person and a second set of characteristics associated with the face of the person at the front door whose identity is not yet determined.

Based at least in part on the score, the resident device may then determine whether they are in fact the same person (e.g., whether the person at the front door is the same as the personal contact in the photo library). The resident device (HomePod mini) may then provide a notification based at least in part on the determination. For example, the resident device may transmit the notification to the user device that the person at the particular location is (or is not) the same as the first person, who was identified as a contact associated with the user device.

In an illustrative example, consider a scenario in which a resident device within a home environment provides notifications about the presence of a person nearby (or inside) the home. In this example, the resident device may be a home automation device (e.g., a smart speaker, a smart digital media player) that is communicatively connected to a camera.

In one example, the camera may be set-up to observe the area around the front door of the home (e.g., to capture images of people who may knock on the door and/or ring the doorbell). Accordingly, the resident device may be configured to receive and process one or more images from the observation camera.

In some embodiments, the facial characteristics model may be trained to detect and recognize a face in a situation in which a person is wearing a face mask. For example, consider a scenario in which a person approaches the front door of a home wearing a face mask. In this scenario, the facial characteristics model may recognize that the person is wearing a facemask, and generate a first faceprint of the face of the person wearing the face mask.

The facial characteristics model may be trained to account for the presence of a face mask, and may compare the first faceprint with faceprints generated from images of faces in a reference set of images drawn from the photo gallery of a user device. In some embodiments, although a face may be recognized even with the presence of a face mask, the image of the face with the face mask may not be included within a notification sent to the user device.

Apple's patent FIG. 1 below is a simplified block diagram (#100) that illustrates a system notification service operating in a home environment #101. The home environment may include one or more people who have some affiliation (e.g., family members, roommates, etc.). In this example, user #106 and user #110 may represent affiliated users, and, may respectively be associated with user devices #108 and #112. Also, within the home environment there may be a resident device #102 (e.g., a tablet, a smart home controller, a smart digital media player, a home automation device or a smart TV device).

(Click on image to Enlarge)

Apple's patent FIG. 3 below is another simplified diagram depicting an example process performed by a system. The process #300 is an example high-level process for a system (e.g., resident device #102 of FIG. 1) determining the presence of a person at a location such as a front door. In some embodiments, a first set of characteristics #313 may correspond to a faceprint (or multiple faceprints) generated from the plurality of image croppings #311. A person #317 may correspond to the person #120 of FIG. 1, a camera #315 may correspond to the observation camera #122 of FIG. 1, an image #319 (e.g., which may be one of a plurality of images (or frames)) may correspond to an image captured by camera #315, and a second set of characteristics #321 may correspond to a faceprint generated from the image #319.

This is a completely new concept for Apple and as such, the details are many. Review Apple's patent application number 20210383100 for more.

Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Some of Apple's Inventors

Hendrik Dahlkamp: Machine Learning Manager; HomeKit Secure Video 

Vinay Sharma: AI/ML, Computer Vision, Deep Learning. Human and Object Understanding

Nitin Gupta: Machine Learning Engineer

Floris Chabert: Senior Research Engineer

Jonghoon J.: Deep learning for computer vision

Camera and Visitor User Interfaces

A second patent on this subject matter was published today titled "Camera and Visitor User Interface. Apple's patent FIGS. 6D and 6E below illustrate a front door camera UI that could send alerts to a user while watching Apple TV.

You could review Apple's second patent application 20210383130 here for details.

Activity Zones for Camera Video

A third patent on this subject matter was published today titled "Activity Zones for Camera Video."

Apple states that at present, cameras can be utilized to monitor aspects of buildings. In some instances, the cameras may be embedded in a smart appliance. For instance, cameras may be included in an electronically controllable doorbell to provide video associated with an entity that activated the doorbell. When the camera detects motion it may transmit a notification to a user indicating that motion was detected. When cameras are positioned in view of public spaces (e.g., sidewalks, public streets, etc.) or neighboring buildings (e.g., shared walkways, shared driveways, windows of the adjacent buildings, etc.) users may be inundated with notifications associated with motion in the public places or neighboring buildings that are not relevant to the user.

[0005] A user device may establish an activity zone or an exclusion zone within video representing a physical view captured by the camera. An automatic monitoring system may analyze the video of the camera with respect to the activity zone or exclusion zone and only trigger notifications that are associated with a zone so as to limit notifications transmitted to the user device to those notifications that are relevant to the user. With activity or exclusion zones video from a camera may be automatically monitored to reduce a volume of notifications transmitted to the user device.

Apple's patent FIG. 5 below illustrates an example operation of object detection using exclusion zones.

Review patent application 20210383554 for more details.