To date, Patently Apple has covered 59 patents regarding future in-air gesturing and the vast majority of them have related to controlling a future mixed reality headset UI. There have been the odd one relating to working with Macs, like this one posted in 2020. Today the US Patent & Trademark Office published another patent application from Apple relating to in-air gestures to be used with future Macs.

In Apple's patent background they note that hand movements and other gestures may involve relatively quick movements. Accordingly, gesture recognition systems that use images from traditional frame-based cameras may lack accuracy and efficiency or otherwise be limited by the relatively slow frame rates that such cameras use. Event cameras capture events as they occur for each pixel and may provide the capability to capture data about user movements significantly faster than many frame-based cameras. However, gesture recognition systems that use event camera data may face challenges in attempting to identify the gestures of a user because, in a given physical environment, there may be many events that are unrelated to a gesture that is being tracked. The need to analyze and interpret potentially vast amounts of event data from a physical environment may significantly reduce the ability of such event camera-based gesture recognition systems to quickly, efficiently, and accurately identify gestures.

Apple's invention covers devices, systems, and methods for event camera-based gesture recognition using a subset of event camera data. In some implementations, a gesture is identified based on a subset of event camera data.

In some implementations, the subset of event camera data is identified by a frame-based camera having a field of view (FOV) that overlaps the event camera FOV.

In some implementations, the event camera FOV and the frame-based camera FOV are temporally or spatially correlated. In one implementation at an electronic device having a processor, event camera data is generated by light (e.g., infrared (IR) light) reflected from a physical environment and received at an event camera.

Various implementations disclosed include devices, systems, and methods that identify a path (e.g., of a hand) by tracking a grouping of blocks of event camera events. Each block of event camera events may be a region having a predetermined number of events that occur at a given time. The grouping of blocks may be performed based on a grouping radius, a distance, or time, e.g., blocks within a given 3D distance of another block occurring at an instant in time are included. Tracking a grouping of blocks as the grouping of blocks recurs at different points in time may be easier and more accurate than trying to track individual events moving over time, e.g., correlating events associated with the tip of the thumb at different points in time.

In some implementations, the electronic device is the same electronic device that includes the event camera (e.g., a MacBook). In some implementations, the electronic device is a different electronic device that receives the event data from an electronic device that has the event camera, such as an iMac or HMD. Apple's patent figures focus on a MacBook as presented below.

Apple's patent FIG. 3 is an overview of the system that includes a gesture recognition unit #350; FIG. 6 is a block diagram of an example configuration of a MacBook including an event camera to track a path of an entity (e.g., a hand).

Apple's patent FIGS. 7A-7B above are diagrams illustrating an example of events detected by an event camera and a grouping of blocks of the detected events at a given time.

For more details, review Apple's patent application 20220157083. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Inventors

Sai Harsha Jandhyala: Senior Electrical Engineer- Camera Electronics Group

Raffi Bedikian: Computer Vision Engineer (5 years). Recently left Apple to work for Humane.