On Thursday the US Patent & Trademark Office published a patent application from Apple that relates to a new projection system that could work with a desktop to present an illuminated keyboard or gaming controller; work with a future mixed reality headset to provide a floating keyboard or an interactive gaming object; and finally, be used in a professional application providing an interactive presentation screen or wall.

Apple's invention introduces a selection of concepts relating to a newly designed interactive projection system. The patent covers devices, systems, and methods for detecting a user input to an electronic device in which the user input is by a touch or a press force applied by a user input object (such as a finger or stylus) against an input surface, a motion of the user input object across the input surface, or a gesture by the user input object toward or away from the input surface.

The input surface may be separate from the electronic device itself. One such system is a computer system which projects an image of a keyboard or other input screen onto a desktop.

A user may then press on a location of the projected image to cause an input to the computer system associated with the part of the image associated with that location. As an example, a user may press against a projected "Enter" key to cause the computer system to execute an action.

Such devices, systems, and methods may have as advantages a reduction of associated hardware components, the ability to project the image against a variety of surfaces, and the ability to scale the projected image, among other advantages.

Apple's patent FIG. 1A illustrates an electronic device for projecting an image onto an input surface and receiving user input by user interaction with the input surface.

The projection system (#106) could be used to present users with an illuminated keyboard or gaming controller system, as the projector communicates with a user's computer (MacBook, iMac, gaming device). The patent figure above illustrates an 'ENTER' button that could be touched with a finger or Apple Pencil to activate the system allowing a user to display a file or program on the user's display.

The projected image #104 can be used as either a sole, primary, or secondary user input mechanism for the projector (#106).

In some embodiments, the light (#108) projected from the projector may include both the projected image and a scanning light beam.

The scanning light beam may be emitted by a self-mixing interferometry sensor within the projector. The self-mixing interferometry sensor may be used to detect a touch, press, or other interaction with the projected image.

The scanning light beam may be a laser beam, and may be infrared or ultraviolet light, for example, so as not to be visible to a user.

In some embodiments, a user's positions, gestures, or motions within the three-dimensional (3D) space between the projector and the input surface (desk #102) may be detected using the scanning light beam. The detection of a lift of a finger off the input surface can be used to indicate an end of a scroll command, a zoom command, or another command.

Technically speaking, using an illuminated projected keyboard could be considered a privacy tool. When the projector system is shut off, intruders would have no way of controlling the computer.

The projection system could have a pro version where in it could be used in schools or corporate presentations, as presented in patent FIG. 1B below. Unlike a standard projection system, the instructor is able to interact with the image with touch, a special pointer or Apple Pencil.

The surface could also be spherical. The projection system would initially scan the surface where the images are to be projected and map it out so that the images will be seen wrapping around the visible area of the sphere or 3D shape. Technically, the surface could be designed to support an interactive wall surface.

In another configuration, Apple states that various embodiments of the scanning and distance detection systems may be used as part of an augmented reality or virtual reality (AR/VR) device.

For example, such a scanning and distance detection system could create an image on the goggles of a user headset of an AR/VR system so that the user perceives an input surface suspended a further distance in front of the user. In other words, a floating keyboard or other instrument if part of a game.

Though no physical input surface exists, the user may be able to interact with the system by physically putting a finger or other user input object at a distance in front of the goggles that corresponds to the projected distance to the virtual input surface.

In Apple's patent FIG. 2A below we're able to see scanning system #200 operable to scan an input surface. The scanning may be performed row-by-row, as indicated.

Alternatively, another scan pattern may be used. The scanning may be performed sufficiently rapidly to allow the scanning system to work in concert with video images projected onto the input surface.

The light source may be a laser emitter, and the emitted light beam 204 may be a laser beam. The light beam 204 may pass through a collimating lens 206 for increased working distance range.

Apple's patent FIG. 4B above illustrates how an intervening object, such as a user's finger (#416) or another user input object (e.g., an Apple Pencil) may be detected during a scan operation (#410).

In the scan operation shown, a beam of light from a self-mixing interferometry sensor is directed toward or across the input surface (#412) by scanning one row, such as row at a time, and stepping vertically through all rows.

An initial scan of the field of view in the absence of a user input object or intervening object may allow the electronic device to determine the extent of the input surface within the field of view. Such an initial scan can provide a baseline of distances to locations on the input surface.

Apple's patent FIG. 6 above is a flow chart of a method 600 for obtaining baseline topographies of a user's finger or other user input object.

Apple's patent application number 20200356159 that was published today by the U.S. Patent Office was filed back in Q2 2019. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Apple Inventors

Mehmet Mutlu: Display Optics Design and Architecture Manager

Fatih Cihan: Senior Optical Hardware Engineer