Over the years Apple has had some massive projects supported by a huge body of patents before the invention ever came to market. For example, the one-time iWallet invention that was covered extensively by Patently Apple took years to come to market as Apple's "Wallet" formerly "Passbook." Apple's iPen/handwriting recognition patents project is yet another invention that has a massive body of supporting patents that could be reviewed in our Archives. The sheer number of patents on this project would strongly suggest that this feature will one day become a market reality. It's been rumored for the longest time that the iPad Pro may introduce this feature. Today, the U.S. Patent & Trademark Office published yet another patent application from Apple on this subject matter. At this time it's unknown as to what handwriting and pen related features will finally make it to market. But for now, Apple's latest invention reveals another piece of the puzzle: Their handwriting recognition software architecture will involve a number of very specific processors to make the iPen experience work flawlessly if not magically, whenever it finally arrives.

Apple's Patent Background

With the proliferation of computing devices such as smartphones, tablets, laptops, and desktops, users are enjoying numerous different mechanisms and interfaces for inputting text, including standard keyboards, virtual keyboards, voice recognition applications, and touch-sensitive input areas. Devices that include such touch-sensitive input areas are able to receive handwriting input from a user, which is then recognized by the device as corresponding to text characters. Furthermore, after a user has input several characters in the handwriting area such that the user is unable to input additional characters, the user has to perform a variety of different clearing operations to clear the handwriting area in order to be able to continue to provide input in the handwriting area. These clearing operations can significantly slow down a user when the user is inputting text to an application executing on the device.

Apple's Invention: Continuous Handwriting UI

Apple's invention provides a device (e.g., mobile device such as a smartphone or tablet, desktop, laptop, etc.) that receives handwriting input in a handwriting area and displays recognized text for the input in a text area of the device. Apple's patent FIG. 22 noted below conceptually illustrates a software architecture of a handwriting recognition system.

In some embodiments, the device displays the text area and the handwriting area on the touch sensitive display of the device. In some embodiments, the handwriting area and the text area may be the same area (for example, a user can handwrite in an area and the input is recognized as text within the same area). The text area can be any area where text input can be displayed, e.g., a note-taking application, an email application, and so on.

The handwriting area provides a region where a user can draw or write a handwritten input. The touch-sensitive display can display one or more virtual keys, buttons, or the like, to provide additional functionality.

In some implementations, the additional keys include a delete (or backspace) virtual key for deleting the last input character and re-displaying a previously cleared handwritten input thus giving the user the ability to modify their handwritten input even after it has been cleared from the handwriting area. Apple's patent FIG. 3 illustrates a "delete" virtual key for modifying handwritten input in a handwriting area.

Other keys include a space key for entering white space and/or accepting recognized text, a return key for entering a line break or the like, and a numbers and punctuation keyboard toggle key for toggling between the handwriting area and a virtual keyboard with number and punctuation keys. In some implementations, the UI includes a virtual button for switching amongst the handwriting recognition user interface and virtual keyboards for various languages.

The handwriting area allows a user to continuously input handwritten characters without having to stop in order to erase or clear the handwriting area (e.g., by applying a "clear-surface" operation) of previously input characters. In some embodiments, the handwriting area automatically clears, or removes a subset of the previous handwritten characters in order to enable the continuous handwriting input feature.

In some embodiments, as a user inputs characters from left to right within the handwriting area (though the described features are equally applicable to right-to-left, top-to-bottom, bottom-to-top, etc. handwriting input), the device automatically clears the handwritten input that was previously received towards the beginning (e.g., left-most region) of the handwriting area. Thus, by the time that a user reaches the end (e.g., right-most region) of the handwriting area, the device has already cleared the beginning area of the handwriting area and the user is able to immediately shift back to the beginning of the handwriting area to continue providing handwriting input.

In some embodiments such as Apple's patent FIG. 1 noted below, rather than completely remove characters from the handwriting area, the device partially fades (e.g., by graying out) the handwritten input. That is, as the user inputs characters from left to right, the device partially fades out the handwritten input from the beginning (e.g., left-most region) of the handwriting area. As the user reaches the end (e.g., the right-most region) of the handwriting area, the user can then start back at the beginning, writing over the faded out handwriting. Once the user begins drawing a new character over a faded out character, the device completely removes this character from the handwriting area. In some embodiments, upon detecting active drawing of a new character over a faded out character, the device clears all of the handwriting input, including both faded out and fully displayed characters, from the handwriting area. In other embodiments, the device only clears the faded out characters over which the user has begun drawing new characters, or all of the faded out characters but not the fully displayed characters.

Apple's patent FIG. 1 illustrates an example user interface for entering characters using handwriting recognition on a mobile device.

When a user draws a handwritten input in the handwriting area, the device compares the handwritten input to characters and words in a dictionary, corpus, repository, or the like. For example, the device can compare the handwritten input against English characters and words formed from English characters in an English character dictionary. Characters or words in the dictionary that include an initial character matching or resembling the handwritten input can be identified as candidates. The device can then automatically display the recognized candidate character (i.e., the "best" candidate) inline in the text area as the current input. In some embodiments, the current input is displayed with an underlining (e.g., "ABC" to indicate that the current input corresponds to the underlined character in the text area).

In some embodiments, the functionality of some virtual keys can be activated using gestures. Thus, for example, when the user is using the handwriting area to enter text, the user can perform a gesture on the handwriting area to add a space or delete a character, for example. In some embodiments, the gestures are pre-defined such that they are easily distinguishable by the device from character strokes. In some embodiments, the touch-sensitive display is a multi-touch sensitive display, and the gestures are multi-touch gestures.

This is an in-depth patent filing with a lot of detail. For those interested in reviewing the finer details of this invention, check out Apple's patent application here.

Patent Credits

Apple credits Deborah Goldsmith and Karan Misra as the inventors of patent application 20150193141 which was originally filed in Q3 2014. Considering that this is a patent application, the timing of such a product to market is unknown at this time. It's interesting to note that prior to coming to Apple, Karan Misra worked as an indie software developer and created Qingwen, a Chinese-English dictionary, and iTransliterate, an Indic transliteration tool, for the iPhone.

Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Comments are reviewed daily from 5am to 6pm MST and sporadically over the weekend.