Apple won 53 Patents today covering the use of Hand Gestures in an Autonomous Vehicle and using AR in Trade Show Mapping
The U.S. Patent and Trademark Office officially published a series of 53 newly granted patents for Apple Inc. today. In this particular report we cover two patents. The fist covers a Project Titan invention where in the future, a user will be able to communicate and direct an autonomous vehicle using hand gesturing. The second patent relates to an iPhone or iPad being able to view Augmented Reality to discover what companies are present at a tradeshow and where they could be found on an AR map.
Gesture-based Control of Autonomous Vehicles
Apple's granted patent relates to Project Titan and specifically to systems and algorithms for using hand gestures and other occupant-generated signals for controlling the motion of autonomous or partially autonomous vehicles.
Motorized vehicles which are capable of sensing their environment and navigating to destinations with little or no ongoing input from occupants, and may therefore be referred to as "autonomous" or "self-driving" vehicles, are an increasing focus of research and development. Given the multiplicity of choices that are typically available with respect to vehicle trajectories in real-world environments, occupant input or guidance with regard to selecting vehicle trajectories (without requiring traditional steering, braking, accelerating and the like) may be extremely valuable to the motion control components of such vehicles. However, providing interfaces for such guidance which are intuitive and easy to use may present a non-trivial challenge.
In the bigger picture, Apple's granted patent covers methods and apparatus for gesture-based control of autonomous or semi-autonomous vehicles are described. In at least some embodiments, a method may comprise one or more computing devices detecting that a triggering condition has been met for initiation of a gesture-based interaction session with respect to an occupant of a vehicle.
Detecting that the triggering condition has been met may itself comprise analyzing or matching a particular hand or body gesture made by the individual within an interaction zone (a three-dimensional region near the occupant, whose boundaries may be customizable) in some embodiments.
Other modes of initiating an interaction session, such as using a voiced command, may be used in other embodiments, and combinations of signals of different modalities (e.g., voice, gesture, gaze direction etc.) may be used in some embodiments.
The method may further comprise identifying one or more options for operations associated with the vehicle, which may be of interest to the occupant participating in the session. A wide variety of options for operations may be identified in different embodiments, including for example passing another vehicle, accelerating the vehicle, decelerating the vehicle, parking the vehicle, changing a direction in which the vehicle is moving, or generating a signal detectable outside the vehicle.
The method may include populating a display with respective representations of at least some of the options. In some cases, the options identified may be assigned respective interest scores or relevance scores based for example on contextual information (such as the time of day, the status of various components of the vehicle such as the gas tank or battery, and so on), personal profiles or preferences of the occupant, and the like.
Apple patent FIG. 2 below illustrates an example vehicle environment comprising a plurality of sensors which may collect data that can be analyzed to respond to gestures or other signals from the vehicle's occupants; FIG. 1 illustrates an example system environment in which operations such as trajectory changes of an autonomous vehicle may be guided at least in part via gestures.
Apple's patent FIG. 3a and FIG. 3b below illustrate examples of gestures which may be used to select from among options being displayed for a vehicle's movements; FIG. 6 illustrates an example scenario in which options for vehicle operations may be ranked relative to one another.
For finer details, review granted patent 10,913,463.
Method of Displaying Virtual Information in a View of a Real Environment
Apple's patent FIG. 1 below shows a schematic illustration of an exemplary AR scenario in which an embodiment of the invention may be employed. Particularly, in an example of a possible scenario in which the invention may be used, FIG. 1 below shows a bird's eye view of a trade show floor. The user holds a mobile device 8 to receive virtual information about the different companies and products present at the fair.
The mobile device is a possible implementation of a system for displaying of virtual information in a view of a real environment. However, the invention is not limited thereto; rather any kind of AR system may be used. For example, the system for displaying of virtual information in a view of a real environment may also be a distributed system in which processors, display and/or camera etc. are not contained in a common housing.
The user might now move inside building 2, where a more accurate pose-estimation-method could be used, e.g. adding wireless LAN-localization mechanisms. Now, the uncertainty of the pose determination might be around 10 meters. The pieces of virtual information about the buildings 1, 2 might now disappear and instead pieces of virtual information (e.g. POIs) about the different areas 5, 6 appear. The user can use this information to orient and navigate to the area of their choice.
When reaching area 6, an optical tracking mechanism could be started, which was trained with data of that area, or a QR-code (QR means quick response, which is a commonly known term in the logistics and in the mobile field) or a marker holding position-information could be detected. Now the accuracy changes to 1 meter or less. At this point, virtual product information, e.g. as a 3D-overlay, could appear and be displayed to the user. In this case the virtual information about areas 5, 6 could also stay visible.
Another embodiment of the invention includes providing an information system which is capable of providing, in a first mode, an augmented reality view and, in a second mode, a map-view, wherein the step of displaying at least one of the pieces of virtual information according to the accuracy information of the current pose is performed in the augmented reality view, but not in the map view. As such, the accuracy information per piece of virtual information is considered in the augmented reality view, but not in the map view.
For finer details, review Apple's granted patent 10,916,056.
The Remaining Patents granted to Apple Today