Yesterday, NVIDIA announced that General Motors and NVIDIA to Collaborate on AI for Next-Generation Vehicle Experience and Manufacturing. One of NVIDIA's strengths in the Auto industry is Autonomous Vehicles (01, 02, 03, 04) working with Volvo, Mercedes and more.

In late February 2025, the U.S. Patent Office published a patent application of NVIDIA'S that relates to autonomous, computer-assisted Shuttles, Buses, Robo-Taxis, Ride-Sharing and On-Demand Vehicles with Situational Awareness. For those interested in the field of autonomous vehicles, NVIDIA's patent provides an interesting overview of autonomous systems.

NVIDIA's patent application states that safe, cost-effective transportation for everyone has long been a goal for modern societies. While privately-owned individual vehicles provide significant freedom and flexibility, shared vehicles can be cost-effective, friendly to the environment, and highly convenient.

While shared and on-demand vehicle operation often benefits from a human driver, there are contexts in which autonomous or semi-autonomous operation can be a tremendous advantage. For example, so-called “GoA4” automated train service has been used for some time in London, certain cities in Japan, and certain other places. The train between London's Victoria Station and Gatwick Airport is fully autonomous, meaning the train is capable of operating automatically at all times, including door closing, obstacle detection and emergency situations. On-board staff may be provided for other purposes, e.g. customer service, but are not required for safe operation. Copenhagen and Barcelona operate similarly-fully-autonomous subway trains. Other trains operate semi-autonomously, e.g., a computer system can safely move the train from station to station, but human personnel are still required to control doors, keep an eye out for safety, etc.

However, designing a system to autonomously drive a shared or on-demand vehicle not constrained to a physical rail without human supervision at a level of safety required for practical acceptance and use is tremendously difficult. An attentive human driver draws upon a perception and action system that has an incredible ability to react to moving and static obstacles in a complex environment.

Providing such capabilities using a computer is difficult and challenging. On the other hand, automating such capabilities can provide tremendous advantages in many contexts. Computers never become fatigued or distracted. They can operate day and night and never need sleep. They are always available to give service. With an appropriate sensor suite, they can simultaneously perceive all points outside the vehicle as well as various points within a vehicle passenger compartment. Such computers could allow humans to focus on tasks only humans can do.

Some aspects of the example non-limiting technology herein thus provide systems, apparatus, methods and computer readable media suitable for creating and running autonomous or semi-autonomous shared transportation vehicles such as shuttle systems. “Shuttles” as used herein includes any suitable vehicle, including vans, buses, robo-taxis, sedans, limousines, and any other vehicle able to be adapted for on-demand transportation or ride-sharing service.

Some example non-limiting systems include situational awareness based on machine perception and/or computer vision by a sensor suite that can rival and, in some aspects, even exceed perception capabilities of human drivers.

Such situational awareness in many embodiments includes awareness (a) within the vehicle (e.g., within the vehicle's passenger compartment) and (b) outside of the vehicle (e.g., in front of the vehicle, behind the vehicle, to the left of the vehicle, to the right of the vehicle, above and below the vehicle, etc.). Such situational awareness can be supported by a sensor suite including a wide range of sensors (e.g., cameras, LIDARs, RADARs, ultrasonic, vibration, sound, temperature, acceleration, etc.) and may in some cases be interactive (e.g., the vehicle may interact with passengers within the passenger compartment and also may interact with pedestrians and other drivers).

Some example non-limiting systems include a software suite of client applications, server applications, and manager clients for operating the system on private and public roads. According to some non-limiting embodiments, the shuttle may follow a predefined route, which may be termed a “virtual rail,” which is typically altered or deviated from minimally or only in specific conditions.

The vehicle may generate the virtual rail itself based on stored, previous routes it has followed in the past. The vehicle in some embodiments is not confined to this virtual rail (for example, it may deviate from it when conditions warrant) but to reduce complexity, the vehicle does not need to generate a new virtual rail “from scratch” every time it navigates across a parking lot it has previously navigated. Such a virtual rail may include definitions of bus stops; stop signs, speed bumps and other vehicle stopping or slowing points; intersections with other paths (which the vehicle may slow down for); and other landmarks at which the vehicle takes specific actions. In some embodiments, the vehicle may be trained on a virtual rail by a human driver and/or receive information concerning the virtual rail definition from another vehicle or other source.

However, in some embodiments it is desirable for the vehicle to calibrate, explore/discover, and map its own virtual rail because different vehicles may have different sensor suites. In typical implementations, the vehicle is constantly using its sensor suite to survey its environment in order to update a predefined virtual rail (if necessary, to take environmental changes into the account) and also to detect dynamic objects such as parked cars, pedestrians, animals, etc. that only temporarily occupy the environment, but which nevertheless must be avoided or accommodated.

The shuttle may stop at any point along the route, including unplanned stops requested by an on-board traveler or pedestrians wishing to ride on the shuttle. In other embodiments, the shuttle dynamically develops a “virtual rail” by performing a high definition dynamic mapping process while surveying the environment. In one example implementation.

In some non-limiting embodiments, the system uses a plurality of client applications, including human-machine interfaces (“HMI”), and devices that allow travelers to call for shuttle service, requesting pick-up time, pick-up location, and drop-off location. In non-limiting embodiments, the client applications include mobile applications provided on mobile or portable devices, which may include various operating systems including for example Android and iOS devices and applications and any other mobile OS or devices, including Blackberry, Windows, and others.

In some embodiments, the system further includes a Web-based application or Desktop application, allowing users to summon a shuttle while sitting at their desk, in their home, etc. For example, the system preferably enables travelers to request a shuttle via a mobile app or kiosk terminals. The system preferably includes kiosks with large screen displays for implementing graphical implementations of Web Applications that allow users to summon shuttles and request service.

Once on-board, the passenger is able to interact with the shuttle via an on-board shuttle client-interface application, Passenger UX. In some embodiments Passenger UX includes camera-based feature recognition, speech recognition and visual information, as well as 3D depth sensors (to recognize passenger gestures, body poses and/or body movements).

In some embodiments, the Passenger UX includes interactive displays and audio systems to provide feedback and information to the riders, as well as to allow the riders to make requests. The on-board displays may include standard read-only displays, as well as tablet or other touch-based interfaces. In some embodiments, Passenger UX is able to detect which display device a particular passenger is currently paying attention to and provide information relevant to that particular passenger on that display device.

In some embodiments, Passenger UX is also able to detect, based on perception of the passenger, whether the passenger needs a reminder (e.g., the passenger is about to miss their stop because they are paying too much attention to a phone screen) or does not need a reminder (e.g., the passenger has already left their seat and is standing near the door ready to exit as soon as the door opens).

In some embodiments, the computer system pilots the vehicle and the safety driver gets involved only when necessary, and in other embodiments the safety driver is the primary vehicle pilot and the computer system provides an assist to increase safety and efficiency.

In embodiments with a safety driver, the shuttle preferably includes an AI assistant or co-pilot system, providing multiple HMI capabilities to enhance safety. In preferred embodiments, the assistant or co-pilot includes features such as facial recognition, head tracking, gaze detection, emotion detection, lip reading, speech recognition, text to speech, and posture recognition, among others.

 The shuttle preferably includes an External UX for communicating with the outside world, including third-party pedestrians, drivers, other autonomous vehicles, and other objects (e.g., intelligent traffic lights, intelligent streets, etc.).

In one aspect, the system preferably includes an AI Dispatcher (“AID”) that controls the system, sets and adjust routes, schedules pick-ups, drop-offs, and sends shuttles into and out of service. A system operator communicates with the AI Dispatcher through a Manager Client (“MC”) application that preferably allows the system operator to adjust system parameters and expressed preferences, such as, for example, average wait time, maximum wait time, minimum time to transport, shortest route(s), cost per person mile, and/or total system cost.

The AI Dispatcher considers the operator's preferences, models the system, conducts AI simulations of system performance, and provides the most efficient shuttle routes and utilization consistent with the system operator's preferences.

The AID may perform AI-enabled simulations that model pedestrians, third-party traffic and vehicles, based on the environmental conditions including weather, traffic, and time of day. The AID may also be used as a setup-utility, to determine the optimal location of system stops/stations for deployment, as well as the optimal number, capacity, and type of vehicles for a given system. The AID may be used to reconfigure an existing system or change the system settings and configurations for an existing system over a given timeframe.

The shuttles according to the present embodiment system and method can operate in a wide variety of different lighting and weather conditions, including Dusk/Dawn, Clear/Overcast, Day/Night, Precipitation, and Sunny conditions. Preferably, the system considers time of day, weather, traffic, and other environmental conditions to provide the desired level and type of service to travelers. For example, the system may dynamically adjust service parameters to reduce traveler wait times during inclement weather or night-time or react dynamically to address traffic conditions.

One example aspect disclosed herein provides a vehicle comprising: a propulsion system delivering power to propel the vehicle; a passenger space that can accommodate a passenger; first sensors configured to monitor an environment outside the vehicle; second sensors configured to monitor the passenger space; and a controller operatively coupled to the first and second sensors and the propulsion system, the controller including at least one GPU including a deep learning accelerator that, without intervention by a human driver: identifies a passenger to ride in the vehicle; controls the vehicle to take on the identified passenger; navigates the vehicle including planning a route to a destination; and controls the vehicle to arrange for the identified passenger to leave the passenger space at the destination.

The identifying may use the first sensors to recognize a gesture the passenger makes to signal that the passenger wishes to use the vehicle. The identifying may be based on the passenger operating a mobile user device. The passenger may specify the destination, and the controller may plan a route to the specified destination. The controller may dynamically plan the route and navigate to the specified destination. The first sensors may include a LIDAR sensor array, and the controller dynamically maps the environment around the vehicle using the LIDAR sensor array. A signaling device on an exterior surface of the vehicle may be used to signal intention to pick up the passenger and/or to signal navigation intentions.

The vehicle may comprise a bus, a taxi, a limousine or a shuttle. The vehicle may further comprise plural wheels in frictional contact with a surface, and the propulsion system drives the plural wheels to propel the vehicle across the surface.

The vehicle may include a passenger information confidence display disposed in the passenger compartment, the passenger information display providing confidence to the passenger.

The GPU may provide massively parallel processing and achieves an ISO 26262 level 4 or higher certification. The second sensors may be configured to simultaneously sense activities of multiple passengers within the passenger space.

There are over 75 patent images related to this patent. Below you'll find only a random sampling of the patent graphics. Patent FIG. 27 Below is an example of a Self-driving Shuttle; FIG. 1, Situational Awareness; FIG. 2A: System Overview.  

NVIDIA's FIG. 21 below shows an example Safety Driver UX—I/O; FIG. 22 shows an example Safety Driver UX—Exemplary Master Display Screen; and lastly, FIG. 26E show an example alternative or additional external intention displays on the front and back of the vehicle that also shows warnings to other vehicles.

NVIDIA's patent that was filed in November 2024 and published in late February listed12 inventors.