This week, Apple was granted a patent for an invention that could be integrated into future MacBooks, iMacs, Magic Mouse and Keyboards. The patent describes the use of various types of sensors that could be placed on devices that would allow a user to make hand and finger gestures to control content on a display. On a MacBook, multiple projectors could be added as noticed in our cover graphic.

Apple’s patent describes new features that could possibly apply to future devices such as MacBooks, iMacs and Magic Mouse/keyboard combinations.  The devices could contain position sensor circuitry configured to gather information on movements of the mouse and where the mouse and/or other device is positioned on a surface and may include user finger input, gestures, and other user input.

The outlined devices may be provided with position sensors having a light source that illuminates a portion of a surface that is overlapped by the mouse and having an image sensor configured to determine a location of the electronic pointing device on the surface and/or to measure movement of the electronic device relative to the surface based on captured images of the illuminated portion. Mechanical device position sensors (e.g., encoded wheels that track mouse movements on the surface) may also be used in monitoring movements of the mouse.

In some configurations, three-dimensional gestures may be supplied by a user (e.g., by moving the user's fingers or hand through the air near an electronic device). Three-dimensional gesture sensors in the electronic device may detect this user input using one or more visible light cameras and/or one or more infrared cameras, using capacitive sensors (e.g., capacitive sensors having arrays of capacitive sensor electrodes on a device housing), or using other sensors. The sensors in an electronic device may include optical sensors (e.g., arrays of light-emitting devices and corresponding light-detecting devices), grip sensors, force sensors, ultrasonic sensors, accelerometers and other sensors to detect orientation, movement, and vibrations, and/or using other sensors.

Images such as images with selectable labeled virtual keys and/or other selectable visual elements may be projected onto an area of a surface that is laterally adjacent to an electronic pointing device or that is otherwise in the vicinity of an electronic device. Accelerometers may measure user finger input vibrations that are conveyed to an electronic device through the surface. Haptic output devices may be used in providing haptic feedback through the surface. In some arrangements, sensor circuitry may be provided on a sidewall of an electronic device and one or more overlapping haptic output devices may be used in providing a user's finger with haptic output when finger motion along the sensor circuitry on the sidewall is detected.

A Few of the Listed Inventors

• Richard Huizar: Prototyping Mechanical Engineer at Apple for user experience prototypes and data collection devices
• Ben Jackson: Engineering Manager
• Paul Wang: Senior Manager, Product Design