Today the U.S. Patent and Trademark Office officially granted Apple a patent that relates to the implementation of a virtual keyboard that provides multimodal input for future MacBooks and desktops that includes fingerprint ID, gaze tracking and more. 

Apple notes that in Multitouch 2D sensing, as used as the primary or secondary means to allow a user to control the function and operation of an electronic device, the positions of all fingertips in contact or close proximity to a sensing surface are tracked and recorded. The arrangement of contacts (e.g., chords) and the movement of the contacts (e.g., gestures) at or near the sensing surface are interpreted by the electronic device as commands from the user meant to modify, initiate, or terminate a function performed by the electronic device.

There are a number of independent sensing modalities that when fused with Multitouch chording and movement data provide enhanced performance and use of electronic devices. The sources of independent sensing data fall into several categories: (1) those that measure some aspect of the user's body state, (2) those that measure data from the environment, which could include sensing data from other individuals, and (3) those that measure some aspect of the state of the electronic device. By way of example, the sense data may include, but are not limited to, the fusion of voice, finger identification, gaze vector, facial expression, hand-held device dynamics, and biometrics such as body temperature, heart rate, skin impedance, and pupil size and fingerprint input.

Apple's patent FIGS. 15A and 15B below are diagrams of an electronic device 310, in accordance with one embodiment of the present invention. The electronic device #310 includes a Multitouch (MT) surface #312 for obtaining contact information associated with fingers in contact with the MT surface, and an imaging device (camera) #314 for obtaining images of hand(s) hovering over the MT surface. As shown, the image sensor is positioned above the hands relative to the plane and at a top edge of the MT surface. It is configured to create an image of the hand(s) relative to the X axis.

Furthermore, the image sensor (camera #314) is embedded underneath or recessed behind the top surface of the electronic device. It is configured to image the hand through an optically member #316 disposed at the top surface of the electronic device. In some cases, the optical member may be configured to hide the image sensor (camera) from view as for example behind a bezel or opaque portion. In one embodiment, the image sensor (camera) is angled relative to the top surface (as shown) so that it can image the hands but be hidden underneath a bezel portion #318. For example, it may be angled 30 degrees relative to top surface.

Apple's patent FIGS. 16 and 17 above illustrate a MacBook and iMac as devices that could use a virtual keyboard device.  

For more details, review Apple's granted patent US 11481109 B2.

The original patent filing was made in 2007 and was well ahead of its time. In 2015 Apple filed for another virtual keyboard patent relating to the iMac. In 2021 Apple filed for its most ambitious virtual MacBook surface.