Apple's SVP of Software points out that it takes Computational Power to make Apple Pencil's new 'Scribble' feature work in Real Time
A computer’s ability to read handwriting, then translate it into letters and numbers it can understand, has been a challenge going back decades. Think of the hit-or-miss capabilities of the Windows Transcriber in the early 2000s, or the PalmPilot in the late ’90s. Handwriting is so nuanced that just analyzing a static letter’s shape doesn’t work.
Alexander George, noted in his Popular Mechanics report that "Apple, it seems, has found a solution. In the newest update to iPadOS, when you write with the Apple Pencil, the iPad can understand your scrawl and, with Scribble, convert it to typed text.
It works like most machine learning—examples inform rules that help predict and interpret a totally new request—but taps into a smarter data set and greater computing power to do what had stumped generations of previous machines.
While Alexa and Siri rely on a connection to faraway data centers to handle their processing, the iPad needs to be able to do all that work on the device itself to keep up with handwriting (and drawing—machine learning also helps the Notes app straighten out an imperfect doodle of a polygon, for example). That takes way more effort than you’d think."
Craig Federighi, senior vice president of software engineering at Apple told Alexander George that "When it comes to understanding [handwriting] strokes, we do data-gathering. We find people all over the world, and have them write things. We give them a Pencil, and we have them write fast, we have them write slow, write at a tilt. All of this variation."
That methodology is distinct from the comparatively simple approach of scanning and analyzing existing handwriting. Federighi noted that for Apple’s tech, static examples weren’t enough. They needed to see the strokes that formed each letter. "If you understand the strokes and how the strokes went down, that can be used to disambiguate what was being written."
(Click on image to Enlarge)
That dynamic understanding of how people write means Apple’s software can reliably know what you’re writing as you’re writing it, but combined with data on a language’s syntax, the iPad can also predict what stroke or character or word you’ll write next.
The massive amount of statistical calculations needed to do this are happening on the iPad itself, rather than at a data center.
Federighi added that "It’s gotta be happening in real time, right now, on the device that you’re holding. Which means that the computational power of the device has to be such that it can do that level of processing locally." For more, read the full Popular Mechanics report.
Apple's Scribble was introduced this year at WWDC-20. Below is the part of the keynote where both Craig Federighi and Jenny Chen, Apple Pencil Software Engineer explain the new iPadOS Scribble feature. The video is set to begin when the subject actually begins to unfold.
While Scribble's magic is tied to iPadOS 14, Apple's webpage introducing the new iPad Air, points out that their new A14 Bionic processor added support for Apple Pencil. Whether that means that the new Scribble feature will work better on iPads using the A14 Bionic is unknown at this time.
For more about what you'll be able to do with Scribble, read Apple Support's webpage titled "Draw, annotate, and use Scribble in Pages."