The Galaxy Note with S Pen is a distinct smartphone niche within the Galaxy world of smartphones. The S Pen works with a limited set of Samsung apps that pop up in Air Command, a semi-circle interfaces as is shown below. There a few videos (one & two) reviewing these limited apps.

While Apple has a few patents on Apple Pencil used with a future iPhone (one and two), it's still a work in progress. TrendForce took a wild leap earlier this month to say Apple Pencil would be used with one or more 2018 smartphones but analyst Ming-Chi Kuo has now stepped in to say that the Apple Pencil option for iPhones is a no go for this year because it delivered a poor experience.

Last week the U.S. Patent Office published a patent application from Microsoft that wants to change the pen and touch experience on both smartphones and tablets. They've invented a method and system to advance a display to accept what they call bi-manual input so that users will be able use more professional apps with touch displays in the future.

Microsoft introduces this in-part as "Thumb + Pen" inputs that will be able to work on a variety of real-world apps like Microsoft Office, far beyond scribbling notes in a special isolated app. Overall, the patent covers their "Unified System for Bi-Touch Interactions."

For instance, the actions of the thumb can be used to determine how pen actions made using the other hand are interpreted. In other aspects, the pen can indicate an object through pointing, while the thumb indirectly manipulates one or more of its parameters through touch controls.

Marking menus, spring-loaded modes, indirect input, and conventional multi-touch interfaces are applied with respect to the bimanual input mode in which one hand is positioned to hold or support the device, and the other hand is free.

The functionality of the user's device is improved by the invention in at least that an improved user experience is provided that enables the user to manipulate virtual objects displayed on the device with bimanual input while one hand is otherwise engaged in manipulating the physical object of the device.

The range of inputs available to the user via the combined thumb of the engaged hand and the input device of (or held by) the free hand are greater than those available to either alone, and thus the efficiency at which the user can manipulate content is improved, and fewer computer resources are expended in effectuating user commands.

Microsoft's patent FIG. 29 below illustrates one example of the architecture of a system for rich bimanual interactions for touchscreen enabled devices

Content developed, interacted with, or edited in association with the unified system for bi-touch interactions #130 is enabled to be stored in different communication channels or other storage types. For example, various documents may be stored using a directory service #2922, a web portal #2924, a mailbox service #2926, an instant messaging store #2928, or a social networking site #2930. The unified system for bi-touch interactions is operative to use any of these types of systems or the like for rich bimanual interactions for touchscreen enabled devices.

Microsoft's patent FIGS. 28A and 28B illustrate a mobile computing device #2800, for example, a mobile telephone, a smart phone, a tablet personal computer, a laptop computer, and the like that could use the bi-manual inputs.

Thumb + Pen Interactions

Microsoft's patent FIGS. 2A-2B below are illustrations of example graphical user interfaces that enable thumb + pen interactions. According to one aspect, the thumb + pen interactions are utilized to provide the benefits of simultaneous bimanual pen and touch interactions. More specifically, the thumb interacts with the touchscreen in an indirect manner, which complements the direct input provided by the preferred hand.

The technique allows users to directly write, draw, and point with the stylus, or perform standard multi-touch gestures such as pinch-to-zoom, panning, or dragging, all with the preferred hand, while the thumb of the non-preferred provides spatially distinct touch input that holds down spring-loaded modes, manipulates parameters, or switches to different tools.

In another example, FIG. 2B illustrates that the roles may be reversed, namely that the preferred hand hovers over the target and the thumb of the non-preferred hand manipulates the pen target.

Patent FIGS. 3B,E & G are illustrations of example graphical user interfaces that enable thumb + pen interactions.

In FIG 12B below you could see the pen tools widget displayed in a spring-loaded mode. The items within the spring-loaded mode items list include options for ink, copy, drag, text and edit modes. Once a tool or function has been chosen, such copy text, the pen could copy and drag that text to desired cell within an app such as excel.

In one example, the functionality may be performed in two strokes comprising a first stroke to select cells, thumb tap to copy, a second stroke to paste cells. In another example, the functionality may be performed in one stroke, including pen down to select cells, thumb tap to copy, pen move and lift up to paste. Each of these methods may be beneficial based on the circumstance.

In Patent FIG. 24B below, we see the object location includes a layer control that allows users to move their design objects to the fore and back. This mode works particularly well in combination with dragging objects with the pen. The pen directly moves an object in 2D space, while the thumb indirectly controls the third dimension in form of layers. The outcome is a compound technique that integrates both pen and thumb to a unitary whole, that can be equally employed for any 3D object translation task.

Being able to interact with a tablet using both touch and pen in concert could be a big step forward and far ahead of anything Samsung or Apple has done to date. Of course it's a patent application and not feature available today. But it does give us hope that Silicon Valley is trying to intelligently advance productivity and add distinct value to smartphones, tablets and 2-in-1's that offer a touch screen.

Back in March Macworld pointed to a rumor from China noting that "It's desperately vague at the moment, but word on the street suggests Apple is focusing on expanding the range of applications with optimized support for the Apple Pencil."

In the end, mobile devices that support the stylus/digital pen input have been limited to a few note-type apps and a few artsy-apps. The Surface works with Adobe's Photoshop and that app will be coming to the iPad in the not-too-distant future. Yet upgrading the stylus to work with popular productivity apps is lacking and Microsoft's patent shows us that some of the best engineers on the planet are trying to change that. Here's to hoping that Apple is working on this too.

About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.