Apple wins a patent relating to future autonomous vehicle cameras being used by Law Enforcement to find people or vehicles of Interest
Today the U.S. Patent and Trademark Office officially granted Apple a Project Titan patent that relates to computer systems for autonomous analysis of sensor data. The camera systems on future Apple vehicles may be tapped into by law enforcement around the world in assisting them to track vehicles or people that could be a threat to security in the modern urban and suburban environments. While it's a fascinating concept to facilitate law enforcement, alarms are likely to be set off by privacy advocates.
Sensor data analysis may be required for a variety of applications. For example, given the challenges and continuous threat to security in the modern urban and suburban environments, video-based surveillance is vital to obtain evidence and could be used to facilitate real-time on-the-fly responses to emergency events.
In 2014, over 245 million video surveillance cameras were installed globally. Sensor data obtained from a variety of sensor types including video and LIDAR (light detection and ranging) sensors may have to be analyzed for making decisions regarding future movements of autonomous vehicles, which are increasingly a focus of research and development.
Depending on the particular application, a variety of types of objects may be of interest—for example, in the video-based security environment, individuals who may be performing potentially harmful activities may be of interest, while in the environment of an autonomous vehicle, other vehicles, pedestrians, road signs and the like may be of interest. Identifying, tracking and predicting future states of objects of interest in a variety of application domains using sensor data remains a challenging technical problem.
According to Apple's granted patent, in some embodiments, the techniques described in the patent may be applied at least in part to video data, which may for example be captured from surveillance cameras, autonomous vehicles, and the like.
In applications using recorded video from surveillance cameras for evidence collection or post-analysis, several aspects of the analysis may be considered important. Firstly, to the extent possible, the camera should readily capture important and relevant data when an abnormal event is triggered, irrespective of the time of the day the event takes place.
Secondly, to the extent possible, the resolution and quality of the captured data must be sufficiently high in order to be of use to investigators after the fact. Due to storage and other hardware costs, many security cameras may not be high definition, but rather may capture video in low resolution, which may lose fine details. This may for example lead to a problematic situation where a suspect/perpetrator's face is blurred or the details of an object of interest (e.g., a license plate) is too grainy to be usable for later investigations.
Accordingly, in at least some embodiments, an intelligent system for sensor data analysis may utilize deep neural networks to dynamically focus on the objects-of-interest at the appropriate times by controlling a dynamic camera with pan, tilt and optical zoom capabilities. Such an intelligent system may be referred to in some embodiments as an attention-focusing system.
In at least one embodiment, sensor data collected using various types of sensors of an autonomous or semi-autonomous vehicle may be analyzed using deep neural networks to enable decisions regarding the movements of the vehicle to be made.
Objects of interest in the operating environment of the vehicle (such as a road on which the vehicle is moving, other vehicles in the vicinity, pedestrians etc.) may be identified and tracked over successive frames of video or other sensor data in various embodiments.
Future movements of at least some of the objects may be predicted using the results of the tracking. An on-board computer system of the vehicle may then use such information to make a variety of decisions, include planning the movements of the vehicle, issuing low level motion directives to motion controllers, and so on.
Some vehicles may include a number of video cameras that can be used to monitor the vehicle's surroundings on an ongoing basis in various embodiments. However, at least during some intervals of time, the captured video data may not include anything of interest that requires further analysis. In such a setting, the vehicle may monitor the captured video data to check for objects of interest.
In various embodiments, one or more neural network-based models may be used to obtain unique representations for each object-of-interest (e.g., a person, a person's face, a car, a road sign, or any other object-of-interest).
Apple's patent FIG. 1A below illustrates an example computing system in which objects of interest in sensor data may be analyzed using deep neural networks; FIG. 2 illustrates an example flow of processing for detecting a region of interest in video data.
Apple's patent FIG. 8 illustrates an example flow of processing for storing and retrieving data to and from a region of interest (ROI) identity database; FIGS. 9A and 9B illustrate examples of scheduling the movements of a camera based on detected objects and object velocities; and FIG. 10A illustrates an example flow of processing for determining attention focus of one or more cameras.
It's unknown whether this project was initiated by Apple or in conjunction with law enforcement, but all three inventors listed on the patent, including Apple's former Director of AI Research, Ruslan Salakhutdinov, are no longer at Apple.