One day, Writing on an iPad will feel like real paper or using a Brush Tool will feel like you're Painting on a Real Canvas
Today the US Patent & Trademark Office published a patent application from Apple that relates to future advanced haptics for Apple Pencil that will provide users with the sensation of writing or drawing on textures. This isn't the first Apple Pencil patent focused on delivering texture haptic feedback. In fact Apple's work on this project was first made public with a pair of patents in the summer of 2015.
The first patent report that we posted in late July 2015 was titled "Apple Invents an Incredible Stylus with Texture Sensing Capabilities, 3D Image Generation & More," which is similar in many ways to the one published today.
The second patent report was titled "Second Apple Stylus Invention Surfaces Regarding Haptics that could Simulate Texture." Apple's engineers had really gone all out this invention. One of the patent figures from that patent is presented below.
In Apple's patent FIG. 11 shown below we're able to see a haptic feedback device which may be, for example, a rumble pack #29 such as is known in the electronic gaming arts could be used in conjunction with a stylus to generate haptic feedback to a user.
An image, such as texture image associated with cloth #23 is captured by the camera stylus. That captured image is displayed on the screen of a tablet. The stylus is held by the user in one hand while the rumble pack is simultaneously held in another hand.
As the stylus is passed over the tablet's screen the image is sensed by the stylus and the corresponding texture information is conveyed to rumble pack by software associated with the tablet. The user is thus able to visually observe the image on the tablet's screen while feeling the texture of the displayed image through rumble pack. The combination of visual and tactile input to the user allows them to experience the "look and feel" of a fabric.
Today's patent application isn't as exotic as what we covered back in 2015 but does envision an improved Apple Pencil with advancements that will allow users to experience advanced haptics that will finally provide users with the ability to feel texture in sync with graphics on the display of an iPad for instance.
For example, the haptic feedback can be provided to the Apple Pencil tip to render texture sensations to simulate drawing on a textured surface. By providing haptic feedback at the tip rather than generally across the entire stylus, the haptic feedback can more accurately mimic the sensation of a writing instrument on a textured surface.
Both the input functions and the haptic feedback functions can be performed based on movement of the tip. As such, the same tip that is used to provide inputs can receive haptic feedback during use.
Apple's patent FIG. 3 below illustrates a block diagram of Apple Pencil, with patent FIG. 5 illustrating a partial sectional view of the stylus with new components and systems.
Apple's reference to texture feedback for Apple Pencil stems from their Haptics feedback system detailed below.
The Haptic Feedback System
Cutting to the chase, the heart of the patent is its haptic feedback system #150 of FIG. 5 above. It's able to provide haptic feedback to a user by moving the Apple Pencil tip #190 relative to the housing #110.
In contrast to haptic feedback applied directly to the housing, haptic feedback provided at the tip more directly provides sensations relating to the tip. For example, haptic feedback can be provided while the tip is applied to a surface of say, an iPad. As the tip moves over the display's smooth surface, the movement of the tip relative to the housing simulates the sensation of moving an instrument over a textured surface.
Accordingly, the haptic feedback can render texture sensations to simulate drawing on a textured surface with the stylus. Such sensations can be more desirable to a user who is familiar with working on a textured surface.
The parameters of the haptic feedback can be selected based on the texture that is desired to be simulated. For example, different types of surfaces (like paper, a canvas, etc.) can be simulated differently based on preprogrammed and user-selectable profiles.
By further example, different types of instruments (e.g. pencil, ball-point pen, fountain pen, highlighter, brush, etc.) can be simulated differently based on preprogrammed and user-selectable profiles.
The texture sensations can be altered based on operation of the Apple Pencil. For example, the haptic feedback can be provided when the tip is applied to and moving along the surface of a tablet. The force of the contact, the speed of the stylus, the orientation of the stylus, and/or the textured surface to be simulated can be considered to determine the operation of the haptic feedback system.
Other Future Apple Pencil Features
Apple's patent filing also looks to other future features for Apple Pencil beyond feedback for textures.
Apple notes that other future components could include, without limitations, adding sensors to detect environmental conditions and/or other aspects of the operating environment of Apple Pencil such an ambient light sensor, proximity sensor, temperature sensor, barometric pressure sensor, moisture sensor, motion sensor (such as an accelerometer, a gyroscope, a global positioning sensor), a tilt sensor, and so on.
Additionally, Apple Pencil may one day be able to detect biological characteristics of the user manipulating the stylus with a biosensor that detects skin temperature, heart rate, respiration rate, blood oxygenation level, blood volume estimates, blood pressure, or a combination thereof.
Apple Pencil will be able to estimate, quantify, or estimate a property of an object nearby or otherwise external to the pencil with a utility sensor such as magnetic field sensors, electric field sensors, color meters, acoustic impedance sensors, pH level sensor, material detection sensor, and so on.
It's also possible, according to the patent filing that Apple Pencil will include additional hardware features such as a micro speaker, a microphone, an on/off button, a mute button, a biometric sensor, a camera, a force and/or touch sensitive trackpad, and so on. And where's the portable missile launcher feature?
Apple's patent application 20190384402 that was published today by the U.S. Patent Office was filed back in Q2 2018. Considering that this is a patent application, the timing of such a product to market is unknown at this time.