Patents show Apple & Microsoft Racing to bring Hand Gesturing Systems to Computers, Home Appliances & more
In late June Patently Apple posted two back to back patent reports titled "Apple Working on a Future Hand Gesturing Interface to Control Gaming, Apple TV and Home Appliances," and "Apple Invents a 3D Depth Mapping Camera for Hand Gesturing Interfaces for Future Macs & Smartglasses." Late last week the U.S. Patent Office published a Microsoft patent covering an advanced hand gesturing system that will one day work with home appliances, desktops, notebooks, vehicles and so forth. The patents clearly show us that both tech giants are investing a lot of time and money into developing highly sophisticated next-gen natural user interfaces (NUIs) for a wide range products such as home appliances, infotainment systems, mobile devices and desktops alongside Digital Assistants and traditional operating systems.
While Microsoft used to rely on technology derived from an Israeli company called PrimeSense for their hand gesturing system linked to their Kinect device, Apple acquiring the company disrupted Microsoft's gesturing roadmap. Microsoft then hired another Israeli team led by Kfir Karmon to lead their new "Project Prague" hand gesturing system (Technology, Research, SDK & Experiences) that allows adding custom hand gestures to any application or device in little to no lines of code.
Kfir Karmon and his team at Microsoft filed for a patent titled "Electrical Device for Hand Gestures Detection." Today's report presents you with a brief overview of their invention.
Microsoft's Patent Background
With the evolution of computerized environments, the use of human-machine interfaces (HMI) has dramatically increased. A growing need is identified for more natural human-machine user interface (NUI) methods such as, for example, voice and/or gaze and more specifically for hand gestures interaction to replace and/or complement traditional HMIs such as, for example, keyboards, pointing devices and/or touch interfaces. Doing so may serve to, for example, eliminate and/or reduce the need for intermediator devices (such as keyboard and/or pointing devices), support hands free interaction, improving accessibility to population(s) with disabilities and/or provide a multimodal interaction environment. Current solutions for identifying and/or recognizing hand(s) gestures may exist, however they are mostly immature, present insufficient accuracy and/or high complexity while requiring high computation resources for extensive computer vision processing and/or machine learning. Integration of such solutions into existing and/or new products, systems, platforms and/or environments may present major challenges which may not be easily encountered and may preventing such solution from being adopted for wide scale usage.
Microsoft's invention relates to an electrical device for detecting hand gestures of a user by estimating a runtime sequence of runtime hand datasets through analysis of one or more images depicting movement of hand(s) of a user with respect to a plurality of pre-defined hand gestures to identify the runtime sequence as a valid hand gesture.
The electrical device architecture may be based on hardware and/or a combination of hardware executing software instructions. The electrical device may connect to one or more imaging devices, for example, a camera, a stereo camera, an infrared (IR) camera and/or a depth camera which monitors a moving hand of a user to receive one or more timed images depicting the moving hand.
Using the electrical device to detect the hand gestures may present major advantages with respect to integration of gesture detection capabilities in existing and/or new high level devices, products, systems platforms and/or solutions.
The electrical device may enable the hand gesture HMI (human-machine interfaces) for a plurality of products, applications and systems, for example, internet of things (IOT), smart home, gaming, learning, medical, sports appliances, automotive, customer service, smart conferencing, industrial applications and the like.
In the patent figures 4, 6 and 8 below you can see three distinct pre-set hand gestures used in a future Microsoft gesturing system with devices mentioned in the paragraph above.
Microsoft's example FIG. 10 below shows how a future Microwave system could be controlled by hand gestures. Gesture 750E is to signal start cooking; gesture 750F is to signal turn dial to increase time in microwave timer; and gesture 750G is to stop the Microwave cooking time.
Microsoft also notes that their system could be extended into a vehicle system. Microsoft notes in patent point #54 "The motion location feature may be adapted for the environment and/or the purpose of an electrical device, for example, a laptop add-on electrical device, a car control unit, a home appliance control and the like. In one case the electrical device is used and/or integrated in a car, the one or more location reference objects may be, for example, a steering wheel, a gear stick and/or a dashboard."
The range of products that Microsoft points to goes far beyond their previous work with their "Kinect" device.
Now with 3D depth cameras being small enough to begin entering future Apple and Microsoft smartphones, tablets and products through to the smart home and vehicles, it's clear that the race is on to deliver AI and hand gesturing systems as part of the movement towards natural human-machine user interfaces (NUIs).
For more details, see Microsoft's patent application 20170192513 that was published by the U.S. Patent Office this past week and originally filed in December 2015. Considering that this is a patent application, the timing of such a product to market is unknown at this time.
Patently Mobile presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. About Posting Comments: Patently Mobile reserves the right to post, dismiss or edit any comments. Those using abusive language or behavior will result in being blacklisted on Disqus.