Earlier today Patently Apple posted a patent report titled "New Apple Invention Covers Inspection and Primal Neural Networks for Controlling Future Autonomous Vehicles." In Apple's second patent application on this subject matter they note that their invention generally relates to systems and algorithms for machine learning and machine learning models. In particular, the disclosure describes a neural network configured to generate output for multiple inference tasks. An autonomous vehicle's multitasking may for example cover changing lanes, observing pedestrians, road segments, lanes, lane markings, staying in a current lane also known as the ego lane, motion control and much more.

Apple's second invention regarding Autonomous Vehicles covers methods, systems and/or techniques for building and using a multitask neural network that may be used to perform multiple inference tasks based on an input data. For example, for a neural network that perform image analysis, one inference task may be to recognize a feature in the image (e.g., a person), and a second inference task may be to convert the image into a pixel map which partitions the image into sections (e.g., ground and sky).

The neurons or nodes in the multitask neural network may be organized into layers, which correspond to different stages of the inferences process. The neural network may include a common portion of a set of common layers, whose generated output, or intermediate results, are used by all of the inference tasks.

The neuron network may also include other portions that are dedicated to only one task, or only to a subset of the tasks that the neural network is configured to perform. When an input data is received, the neural network may pass the input data through its layers, generating outputs for each of the multiple inference tasks in a single pass.

In some applications, the ability to efficiently make multiple inferences from a single sample of input data is extremely important. As one example, a neural network may be used by an autonomous vehicle to analyze images of the road, generating multiple outputs that are used by the vehicle's navigation system to drive the vehicle.

The output of the neural network may indicate for example a drivable region in the image; other objects on the road such as other cars or pedestrians; and traffic objects such as traffic lights, signs, and lane markings. Such output may need to be generated in real time and at a high frequency, as images of the road are being generated continuously from the vehicle's onboard camera. Using multiple independent neural networks in such a setting is not efficient or scalable.

The multitask neural network described herein increases efficiency is such applications by combining certain stages of the different types of inference tasks that are performed on an input data. In particular, where the input data for the multiple inference tasks is the same, a set of initial stages in the tasks may be largely the same. This intuition stems from the way that the animal visual cortex is believed to work. In the animal visual cortex, a large set of low level features are first recognized, which may include areas of high contrast, edges, and corners, etc. These low-level features are then combined in the higher-level layers of the visual cortex to infer larger features such as objects. Importantly, each recognition of a type of object relies on the same set of low level features produced by the lower levels of the visual cortex. Thus, the lower levels of the visual cortex are shared for all sorts of complex visual perception tasks. This sharing allows the animal visual system to work extremely efficiently.

This same concept may be carried over to the machine learning world to combine neural networks that are designed to perform different inference tasks on the same input. By combining and sharing certain layers in these neural networks, the multiple inference tasks may be performed together in a single pass, making the entire process more efficient and faster. This is especially advantageous in some neural networks such as convolution image analysis networks, in which a substantial percentage of the computation for an analysis is spent in the early stages.

In addition, the multitask neural networks described herein may be more efficiently trained by using training data samples that are annotated with ground truth labels to train multiple types of inference tasks. The training sample may be fed into a multitask neural network to generate multiple outputs in a single forward pass. The training process may then compute respective loss function results for each of the respective inference tasks, and then back propagate gradient values through the network. Where a portion of the network is used in multiple tasks, it will receive feedback from the multiple tasks during the backpropagation. Finally, by training the multitask neural network simultaneously on multiple tasks, the training process promotes a regularization effect, which prevents the network from over adapting to any particular task. Such regularization tends to produce neural networks that are better adjusted to data from the real world and possible future inference tasks that may be added to the network. These and other benefits of the inventive concepts herein will be discussed in more detail below, in connection with the figures.

Apple's patent FIG. 1 below is a diagram illustrating portions of a multitask neural network; FIG. 2 is a diagram illustrating portions of the multitask neural network to perform image analysis tasks.

Apple's patent FIG. 4 presented above illustrates an example autonomous vehicle using a multitask neural network to analyze road images.

Apple's patent application 20180157972 was filed in Q4 2017 and published in June 7, 2018. For other patents on this subject, see our Autonomous Vehicle Archives.

About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.