There have been many articles on the subject of what's holding back autonomous vehicles from coming to market en masse by The Atlantic, The Conversation, VentureBeat, Forbes and others. Autonomous vehicles are designed with crazy complex systems that make life and death decisions in real-time.

The Atlantic, on sensors alone states: "Autonomous cars use a broad set of sensors to “see” the environment around them, helping to detect objects such as pedestrians, other vehicles and road signs. Cameras help the car to view objects. Lidar uses lasers to measure the distance between objects and the vehicle. Radar detects objects and tracks their speed and direction.

These sensors all feed data back to the car’s control system or computer to help it make decisions about where to steer or when to brake. A fully autonomous car needs a set of sensors that accurately detect objects, distance, speed and so on under all conditions and environments, without a human needing to intervene.

Lousy weather, heavy traffic, roads signs with graffiti on them can all negatively impact the accuracy of sensing capability. Radar, which Tesla uses, is less susceptible to adverse weather conditions, but challenges remain in ensuring that the chosen sensors used in a fully autonomous car can detect all objects with the required level of certainty for them to be safe.

To enable truly autonomous cars, these sensors have to work in all weather conditions anywhere on the planet, from Alaska to Zanzibar and in congested cities such as Cairo and Hanoi."

The Atlantic report then turns to Machine Learning: "Most autonomous vehicles will use artificial intelligence and machine learning to process the data that comes from its sensors and to help make the decisions about its next actions. These algorithms will help identify the objects detected by the sensors and classify them, according to the system’s training, as a pedestrian, a street light, and so on. The car will then use this information to help decide whether the car needs to take action, such as braking or swerving, to avoid a detected object.

In the future, machines will be able to do this detection and classification more efficiently than a human driver can." And this is where one of today's granted patents from Apple comes in.

Today the U.S. Patent and Trademark Office officially granted Apple a patent that relates to Project Titan that's focused on the creation of an autonomous electric vehicle.  More specifically, the patent relates to Machine Learning systems and algorithms for reasoning, decision making and motion planning with respect to controlling the motion of autonomous or partially autonomous vehicles.

Motorized vehicles which are capable of sensing their environment and navigating to destinations with little or no ongoing input from occupants, and may therefore be referred to as "autonomous" or "self-driving" vehicles, are an increasing focus of research and development. Until relatively recently, due to the limitations of the available hardware and software, the maximum speed at which computations for analyzing relevant aspects of the vehicle's external environment could be performed was insufficient to enable non-trivial navigation decisions to be made without human guidance. Even with today's fast processors, large memories, and advanced algorithms, however, the task of making timely and reasonable decisions (which are based neither on excessively pessimistic assumptions, nor on excessively optimistic assumptions) regarding an autonomous vehicle's trajectory in the context of unpredictable behaviors of other entities (such as other drivers or other autonomous vehicles) and incomplete or noisy data about static and dynamic components of the vehicle's environment remains a significant challenge.

Various embodiments of methods and apparatus for evaluating varying-size action spaces for autonomous vehicles using neural network-based reinforcement learning models.

According to some embodiments, a method may comprise determining a representation of the current state of the environment of an autonomous or partially-autonomous vehicle at various points in time during a journey.

The method may also include identifying, with respect to the current state, a corresponding set of feasible or proposed actions which can be undertaken. For example, in one embodiment, data captured using various sensors of the vehicle may be processed locally at a behavior planner decision-making component of the vehicle and combined with map information to determine the current state and the feasible actions.

In some states, such as when the vehicle is traveling on a largely-empty straight highway with no turns possible for several kilometers or miles, the number of actions to be evaluated may be relatively small; in other states, as when the vehicle approaches a crowded intersection, the number of actions may be much larger.

Each of the actions may comprise a number of different constituent attributes or properties in various embodiments, such as an origin/source lane segment (the lane segment in which the vehicle is located at the time the actions are evaluated), a target lane segment (the lane segment in which the vehicle will be located if the action is implemented), a target velocity in the target lane segment, positioning relative to one or more other vehicles or objects, and so on.

The decision-making components of the vehicle may be responsible for evaluating the different feasible actions in a given state relative to one another, selecting one of the actions for implementation, transforming the abstract representation of the selected action into a detailed motion plan, and issuing directives to the lower-level motion control subsystems of the vehicle (e.g., the acceleration subsystem, turning subsystems etc.) to implement a trajectory corresponding to the selected action.

In at least some embodiments, multiple instances or executions of a reinforcement learning model may be employed at the vehicle to obtain respective value metrics for the actions, and the value metrics may be used to select the action to implement.

In at least some embodiments, deep neural network-based machine learning models (such as convolutional neural network models trained using graphical or image inputs) may be employed as the reinforcement learning models.

Apple's patent FIG. 1 below illustrates an example system environment in which varying-size action spaces may be evaluated using a reinforcement learning model to help direct the movements of a vehicle.

The term "autonomous vehicle" refers to vehicles for which at least some motion-related decisions (e.g., whether to accelerate, slow down, change lanes, etc.) may be made, at least at some points in time, without direct input from the vehicle's occupants.

In various embodiments, it may be possible for an occupant to override the decisions made by the vehicle's decision-making components, or even disable the vehicle's decision-making components at least temporarily; furthermore, in at least one embodiment, a decision-making component of the vehicle may request or require an occupant to participate in making some decisions under certain conditions.

(Click on image to Enlarge)

Apple's patent FIG. 2 below illustrates an example decision making scenario for an autonomous vehicle, as well as general problem characteristics associated with such decision-making scenarios.

The system has to understand the route the driver is taking, how and when to merge; how and when to take an exit ramp or turn to a different road; understand multiple vehicle speeds to ensure merging is safe; scan road conditions; understand road signs and street lights (stop, caution, go) and so forth. There are a lot of information that has to be processed in real-time.

(Click on image to Enlarge)

Apple's patent FIG. 9 below illustrates examples of state and action properties or attributes for which respective elements may be identified for including in encodings used as input to a reinforcement learning model.

(Click on image to Enlarge)

With respect to a given feasible action, the set of attributes or properties (#973) to be represented in the encodings may include, for example, the origin or current lane segment (#901), the target lane segment (#902), the target velocity or speed (#903), relative positioning options (#904) with respect to one or more other vehicles in the target lane segment, as well as optional superimposed map modifications in the trajectory (#905) in the depicted embodiment above.

For those that are following the evolution of Apple's autonomous vehicle, there is a lot of data in this granted patent that you could feast on, especially engineers in this field.  For more, review Apple's granted patent 11,243,532. You could also check out our Project Titan archives to review the other 116 patents on record.

When you grasp the complexity of creating a fully functional autonomous vehicle with its army of sensors, its complex LiDAR cameras and its array of Machine Learning algorithms, you can appreciate the fact that it will take Apple some time to bring their first vehicle to market.