Over the last eight months a wide range of patent applications covering autonomous vehicles have come to light, with many of them discovered in Europe. In that time frame we also learned that Apple is working with Volkswagen on an autonomous vehicle. Earlier today Patently Apple discovered yet another European Patent application from Apple that was published on Wednesday August 1, 2018.

The detailed patent covers a future augmented reality windshield system which also is known as a Heads up Display. Apple describes every aspect of their AR display system in painstaking detail. It even describes occupants of an autonomous vehicle having a FaceTime session between different vehicles. The system will even be able to detect and make adjustments so that a panicky occupant could remain calmer in the autonomous vehicle.

Apple notes that in many situations, a graphical overlay can be provided on an environment that is perceived through a transparent surface, including a window. A graphical overlay can provide information to an observer, including information which is relevant to one or more portions of the perceived environment.

In some cases, a graphical overlay is used in a vehicle, where a graphical overlay can be perceived by an occupant of the vehicle and provides information relevant to one or more of the vehicle, including vehicle speed, and one or more portions of the environment in which the vehicle is located. Such information can be provided on a transparent surface, including a forward window, also referred to as a windscreen or windshield such that the information can be perceived by an operator of the vehicle, including a driver, near the line of sight of the operator as the operator observes the environment through which the vehicle is navigated, including an oncoming roadway.

In some cases, the graphics displayed should be able to overlay certain objects located in the environment perceived though the transparent surface and can augment, alter, control, information related to the environment which is provided to an observer via the transparent surface. Such graphical overlays can be referred to as an augmented reality display.

The Autonomous Vehicle Navigation System

In Apple's patent FIG. 1 below we're able to see an illustrated vehicle that includes a vehicle navigation system (VNS) #140. In some embodiments, the VNS can autonomously control one or more vehicle control elements #150 to cause the vehicle #110 to be autonomously navigated along one or more driving routes through the environment #100.

In some embodiments, the VNS controls the display of information to one or more occupants of the vehicle via one or more interfaces #130.

At least one of the interfaces can include a transparent surface via which an occupant can perceive one or more portions of the environment and the VNS can control a display of information to the occupant via a graphical overlay on the transparent surface (vehicle windshield) which provides an augmented reality display of one or more portions of the environment perceived by the occupant via the transparent surface.

External Sensors: Apple notes that the vehicle includes one or more sets of external sensors #116 which generate sensor data representations of one or more portions of the external environment.

The external sensors can include one or more of visible light camera devices, infrared camera devices, near-infrared camera devices, light beam scanning devices, ultrasonic sensor devices, audio sensor devices, depth camera devices, radar devices, geographic position detection devices, and some combination thereof.

Sensor data representations generated by one or more of the external sensors can be communicated to the VNS which can generate and maintain a world model of one or more portions of the environment based on the sensor data representations generated by the external sensors.

In some embodiments, the VNS generates and maintains a world model based at least in part upon information received from a remote service, system, device, etc. via one or more communication networks.

Internal Sensors: Apple further notes that the vehicle includes a set of one or more internal sensors #118 which generate sensor data representations of one or more portions of a vehicle interior included in the vehicle, including sensor data representations of one or more occupants (users) of the vehicle.

The internal sensors can include one or more of visible light camera devices, 30 infrared camera devices, near-infrared camera devices, light beam scanning devices, ultrasonic sensor devices, audio sensor devices, depth camera devices, radar devices, some combination thereof, kinematic sensor devices, etc.

Control Elements: The vehicle includes one or more control elements which cause the vehicle to be navigated. Control elements can include one or more of a steering control element, a motor, a throttle control element, a brake control element, an accelerometer, a speedometer etc.

Vehicle Interfaces: The vehicle includes one or more interfaces #130, which can include one or more user interfaces installed in the vehicle with which one or more occupants can interact, one or more communication interfaces which can communicate with one or more remotely located devices, vehicles, services, systems, etc. via one or more communication networks.

Transparent Surfaces: The vehicle includes one or more transparent surfaces #134 via which one or more occupants can perceive one or more portions of the environment from within the vehicle interior, etc. In some embodiments, the VNS can generate, manage, control, etc. one or more graphical displays, including one or more augmented reality displays, which are displayed on one or more particular transparent surfaces in the vehicle.

The Augmented Reality Display Module: As shown, the VNS can include an augmented reality display module #111, also referred to as an augmented reality display system. In some embodiments, the VNS and the module are implemented by separate computer systems.

A Limited Example of the Augmented Reality Display

Apple's patent FIGS. 2A and 2B presented below (in a limited way) illustrate a vehicle comprising a transparent surface which provides an augmented reality display which includes a spatially positioned representation of a speed of the vehicle relative to a local speed limit.

In some embodiments, an augmented reality display provided to an occupant of a vehicle, via being displayed on one or more transparent surfaces of the vehicle, includes a display element which provides a three-dimensional representation of a speed of the vehicle where representation of the speed is spatially positioned, in the environment, based on a difference between the vehicle speed and a local speed limit.

The example shows a driver #210 looking through an AR windshield could see what speed they're driving and what the actual speed limit is so that the driver is informed if the vehicle is traveling at lawful speeds.

The virtual display is designed to follow the road at all times so that the driver's AR screen elements are properly centered to the road ahead of the driver at all times.

FaceTime Vehicle to Vehicle

Another feature buried in the patent filing that Apple is working on involves FaceTime between vehicle occupants and another vehicle. Apple specifically notes that "In some embodiments, an augmented reality display system included in a vehicle enables visual communication between an occupant of the vehicle and a remotely located user, including an occupant of a separate vehicle."

Panicky Occupant Detection

Two weeks ago Patently Apple posted a report about a 'Personal Driving Profile' system for autonomous vehicles that touched on "Monitoring Occupant Stress Levels and Making Adjustments." Today's patent likewise touches on this aspect of drivers and passengers experiencing stress while an autonomous vehicle is in action.

Apple notes that in some embodiments, monitoring a stress level, cognitive load, etc. of an occupant can include monitoring one or more various features of an occupant, including eye motion, body posture, body gestures, body temperature, breathing rate, eye blink rate, head motion, eye motion, heart rate, some combination thereof, etc.

Based on determining that one or more monitored features of an occupant is associated with a determined value which at least meets a threshold level associated with a particular stress level, cogitative load level, etc. the augmented reality display system can determine that a monitored occupant is associated with the particular stress level, cognitive load level, etc.

The augmented reality display system can dynamically adjust the augmented reality display, including which display elements are presented in the display, based on the monitored stress level, cognitive load level, etc. of one or more of the monitored occupants.

Additional Patent Figures

Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Patently Apple presents a detailed summary of patent applications and/or granted patents with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.