While Apple uses Machine Learning to control FaceTime's feature known as "Center Stage," a new patent advances this by using adjustable infrared illumination that could assist in future hand tracking for Vision related devices along with the iPhone, iPad, MacBooks and possibly foldable devices.

Electronic Devices with Illumination Systems

Electronic devices may have sensors for gathering user input and making other measurements. Some electronic devices may use image sensors. As an example, an electronic device with video conferencing capabilities may use an image sensor to capture images of a user during a video call. As another example, a gaze tracking system in an electronic device may use an image sensor to capture images of infrared light glints on a user's eyes and thereby track the direction of the user's gaze. Some electronic devices may have hand tracking systems. In hand tracking systems, image sensors may be used to capture images of a user's hands. By processing these images, the movement of the user's hands and gestures made by the user's hands and/or fingers can be identified and used as a form of user input.

In some ambient lighting conditions, there may be sufficient ambient light available to operate image sensors. For example, in bright ambient lighting conditions, images of a user's hands may be captured satisfactorily. In dim ambient lighting conditions, however, it may be desirable for an electronic device to emit supplemental illumination to ensure that the user's hands remain visible to the image sensors of the device. Supplemental illumination may be provided at visible and/or infrared wavelengths. As an example, when tracking a user's hands in dim lighting, supplemental infrared illumination may be emitted. The presence of the supplemental infrared illumination may help infrared image sensors capture satisfactory hand images. At the same time, the use of infrared illumination, which is invisible to the naked eye, may help avoid disturbing people who are nearby.

To help ensure that excessive power is not consumed during the production of supplemental illumination, electronic devices may be provided with adjustable illumination systems. The angular spread and intensity of the emitted illumination may be adjusted in real time to focus on target objects of interest. For example, when tracking a user's hand, an adjustable illumination system may produce a beam of infrared light that is focused on the user's hand. As the user's hand is moved, the infrared light beam may be steered to follow the hand. With this approach, it is not necessary to illuminate extraneous background objects in a scene. The use of adjustable illumination systems may therefore help conserve power.

Apple's patent FIG. 5 above is a diagram of an illustrative electronic device that is providing illumination to illuminate a target. Device #10 may include adjustable illumination system #32 and sensor #34. The adjustable illumination system may sometimes be referred to as a steerable light source or system, adjustable lighting system, adjustable illuminator, or steerable illuminator, may have one or more light-emitting components such as infrared light source #16 of FIG. 1 that emit light #36 (sometimes referred to as illumination).

Light may illuminate a target object such as target #38. The light may be visible and/or infrared light. The target may be an animate or inanimate object. As an example, the target may include one or more body parts of a user (e.g., a user's hand, fingers, eye, head, full body, etc.) or inanimate objects (e.g., a wand).

When system #32 and sensor #34 are used together as combined system #31 in a head-mounted device (or other device) for hand tracking, this combined system may sometimes be referred to as a hand tracker or hand tracking system.

During hand tracking operations with the hand tracker, hand motions, finger gestures and other hand input may be used as a form of user input for controlling the operation of the device (#10). For example, hand gestures may be used to move virtual objects being presented to the user with the display resources of the device may be used to make menu selections in displayed content, etc. Virtual objects (which may sometimes be referred to as computer-generated content) may be presented in isolation or may be merged with real-world content (e.g., real-time real-world content captured using forward-facing cameras in the housing). 

If desired, virtual hands may be presented to a user in positions that correspond to the real-world positions of the user's hands as sensed with the hand tracker.

For full details, review Apple's patent application 2025007665.

Apple Inventors

• Eddie Huo: Product Design Manager
• Devinne Cullinane: Product Design Engineer - Vision Products

Yesterday, Patently Apple posted another hand tracking patent titled "Apple has filed a patent for Vision-Based Hand Gesture Customization using advanced Machine Learning