Today the U.S. Patent and Trademark Office officially published a patent application from Apple that relates to a new optical touch and/or proximity sensing system, and more particularly, to an optical sensing system that works in conjunction with an optical stylus to determine one or more of the target or touch location, centroid, hover distance, tilt angle, azimuth, and in some instances the orientation and rotation of the stylus.

Photo-Sensing Enabled Display for Stylus Detection

Apple's patent application relates to a new Apple Pencil based on optical system including an optical stylus (Pencil) and optical sensing system that are together operative to detect one or more of the target or touch location, centroid, hover distance, tilt angle, azimuth, and in some instances the orientation and rotation of the optical stylus with respect to the optical sensing system.

In particular, detecting the rotation of a stylus can provide an additional input mode that enables additional stylus functionality. For example, rotating a stylus while interacting with a drawing application can allow the artist to control the perceived texture, thickness or color of a line being drawn.

In some embodiments the optical sensing system is an electronic device having an integrated touch screen with micro circuitry configurable for both display operations and touch/proximity sensing of the object.

In some embodiments, the integrated touch screen can include light emitting diodes or organic light emitting diodes (LEDs/OLEDs), display driving circuitry, and touch sensing circuitry. In some embodiments, the LEDs/OLEDs can be implemented as a micro-LED display including an array of micro-LEDs and micro-driver circuitry.

In some implementations, the array of micro-LEDs and micro-driver circuitry can be configured in a direct current (DC) photoconduction mode to detect the presence of a stylus by detecting unmodulated light transmitted by a stylus.

In other implementations, the array of micro-LEDs and micro-driver circuitry can be configured in an alternating current (AC) photoconduction mode to detect the presence of multiple styluses by detecting modulated light transmitted by the multiple styluses. In still other implementations, the array of micro-LEDs and micro-driver circuitry can be configured in an optical reflective touch mode to detect the presence of an object such as a finger or stylus by detecting modulated light generated by some of the micro-LEDs and reflected off the object.

The detection modes described above rely on light passing through the detection surface of a cover material located above the integrated touch screen to LEDs, OLEDs or micro-LEDs located below. However, light impinging on the boundary between the detection surface and a medium above the detection surface (e.g., air, water, stylus, or finger), from either above or below the detection surface, can reflect off the boundary or be refracted as it passes through the boundary.

In some instances, this reflected or refracted light can be detected and incorrectly identified as an object such as a finger or a stylus. Accordingly, in some embodiments of the disclosure, light illuminator angular filters can be employed above those micro-LEDs configured as illuminators within the integrated touch screen to limit the illumination angle of those illuminators, and/or light detector angular filters can be employed above those micro-LEDs configured as detectors within the integrated touch screen to limit the detection angle of those detectors. These angular filters effectively block or filter light transmitted, reflected or refracted within the cover material to reduce or eliminate the false detection of water droplets on the touch surface.

After the angularly filtered light is detected by the optical sensing system, the resultant illumination pattern (e.g., the target location of a hovering stylus) can be processed to determine the hover distance and tilt angle of the object, and compute various parameters (e.g., the centroid of the illumination pattern representing the target location) and other operations (e.g., stylus tracking) with greater precision.

In some embodiments, the optical stylus is a passive stylus including either a diffusive reflector or retroreflector facets to reflect light emitted from the optical sensing system with a consistent angular reflection profile. Different tilt angles can generate different reflected energy profiles, and these different reflected energy profiles can be evaluated to determine the location, hover distance (if any) and tilt angle of the stylus.

In embodiments, a passive stylus including a diffractive (patterned) reflector can also reflect light emitted from the optical sensing system with a consistent reflected light pattern regardless of an angle of stylus tilt angle with respect to the surface.

Different tilt angles and rotations of the stylus can generate different reflected light patterns, and these different reflected light patterns can be evaluated to determine the location, hover distance (if any), tilt angle, orientation and rotation of the stylus. In some embodiments, a semi-active stylus including an amplitude sensor in its tip and optionally in radial locations along the sides of the stylus can detect the amplitudes of different frequencies of modulated light emitted from the optical sensing system, and detect the location and hover distance (if any) of the stylus, and in some instances also detect the tilt angle and rotation of the stylus.

In some examples, an active stylus including both light emitters and detectors can generate light and receive reflected light when it is reflected off a proximate optical sensing system with a retroreflector layer formed between an array of display elements. Different locations of the stylus on or above the display surface can produce different spectral distributions of reflected light that can be analyzed to determine the location of the stylus.

Enlarge the graphic below to learn more about a few of the key features of this new Optical-based Apple Pencil system.

To review the full details of this invention, check out patent application 20240118773. Considering that Apple only filed for this patent in September 2023, this isn't likely to surface with Apple's new iPads due in May. Though, as the saying goes, Never say Never.