Yesterday the US Patent & Trademark Office published a patent application from Apple relating to Project Titan and more specifically to multi-modal sensing for nighttime autonomous driving object detection and recognition. The superiority of the new system over traditional headlights when driving at night is going to save a lot of lives in the future.

Some automated systems gather process large quantities of sensor data to identify objects in the surrounding environment. The processing of sensor data is often subject to a real-time constraint to facilitate navigation and/or robust control of the automated system. This is what Apple's invention is trying to improve upon.

Overall Apple's invention covers implementations of multi-modal sensing for nighttime autonomous driving object detection and recognition.

Apple explains that nighttime or low-light environments present challenges for automated vehicle control systems. For example, the illumination level provided by headlights on a vehicle at night may be limited by laws or regulations, which may in turn limit the effective range of a visible spectrum sensor (e.g., a camera) used for detecting objects in or near the path of the vehicle.

Having a limited effective range (e.g., about 60 meters = 180 feet) for detecting and or classifying objects can reduce safety and/or reduce the speed at which the vehicle can travel safely.

A combination of multiple complimentary image sensing technologies may be employed to address the challenges of nighttime or low-light environment object detection and classification. For example, there may be looser or no restrictions on the illumination level of a near infrared illuminator mounted on a vehicle.

A near infrared sensor with a near infrared illuminator can be configured to capture high resolution image information about objects in or near a path of the vehicle out to a significantly longer range (e.g., 200 meters = 600 feet) from the vehicle.

This may enable earlier detection and classification of objects as the vehicle moves and improve safety and/or maximum speed. Near infrared illuminators may project near infrared light in a relatively narrow field of view (e.g., a 30-degree cone).

Although their range may be relatively limited, visible spectrum sensors can provide high resolution image data in multiple color channels (e.g., red, green, and blue). Visible spectrum sensors also may provide a wider field of view (e.g., a 120-degree field of view) of the path in front of a vehicle.

Long wave infrared sensors (LWIR) capture naturally occurring thermal radiation from objects in the environment around a vehicle and therefore do not rely on an illuminator. The effective range of a long wave infrared sensor may be limited by the sensor resolution and the resolution requirements for object detection and/or classification.

A long wave infrared sensor, which may include an array of component sensors, may provide a wide field of view around the vehicle (e.g., a 180-degree field of view). Long wave infrared sensors may provide images of objects in the environment that are of relatively low resolution.

In some implementations, objects detected based on low resolution image data from a long wave infrared sensor are classified by adjusting control parameters for other sensing modalities and/or image processing resources to focus computer vision resources of the vehicle on a region of interest associated with the detected objects.

For example, an integration time, an aperture size, a filter, or a gain for a sensor (e.g., a near infrared sensor or a visible spectrum sensor) may be adjusted to enhance a portion of a captured image associated with a region of interest.

For example, a power level or a field of view for an illuminator (e.g., a near infrared illuminator or a visible spectrum illuminator) may be adjusted to enhance a portion of a captured image associated with a region of interest. For example, a computational control parameter (e.g., a resolution used for image processing or a count of image processing passes) may be adjusted and applied to an image portion associated with a region of interest.

The techniques described in Apple's patent application may provide improvements over prior computer vision systems for automated vehicles. Some implementations may increase the effective range at which objects in or near the path of a vehicle may be detected and classified. Some implementations may more accurately classify objects in a low-light environment. Safety of an automated vehicle control system may be improved and/or the maximum safe speed in low-light environments may be increased.

Apple's patent FIG. 4 below is a block diagram of an example of a vehicle configured for multi-modal sensing for nighttime autonomous object detection and recognition; FIG. 7 is a diagram of an example of overlapping fields of view for multiple sensors of different types mounted on a vehicle.

Apple's patent application that was published yesterday by the U.S. Patent Office was originally filed back in Q3 2018. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

News about Project Titan broke back in January about a second Chinese National working on Project Titan being arrested by the FBI for stealing company secrets. Obviously Apple's project has highly valuable autonomous vehicle technologies that the Chinese would love to steal.

About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.