In May 2017 Patently Apple posted a patent report titled "Apple to Advance Apple Pencil to Work with Next-Gen Magic Trackpad for Macs," as presented in the patent graphic below. The patent introduced us to a new inertial sensor mode that would allow Apple Pencil to rotate or tilt drawn content on-screen and even double as a joystick for games.

Over the last few weeks a couple of new Apple patent applications have surfaced regarding possible next-generation versions of Apple Pencil that could go far beyond simply adding Apple Pencil support for the next generation of the Magic Trackpad.

In the first patent, Apple introduces us to the idea that Apple Pencil may one day work on any non-touch surface and have the text or imagery appear on an Apple device such as a Mac (iMac, desktop, MacBook and more).

Apple's latest invention may also allow a future version of the Apple Pencil to create 2D and 3D images and work with augmented and virtual reality editing applications.

According to Apple, their invention relates to "generation of content using an input device without a touch-sensitive surface. In some examples, touch-down and lift-off on a non-touch-sensitive surface can be monitored by a force sensor of the input device. The position and/or motion of the input device can be tracked according to various methods including one or more of a motion and orientation sensor, a camera, or an electromagnetic- or sound-based triangulation scheme. The force data and position/motion data can be processed to generate content, including textual character input and three-dimensional objects. In some examples, the content can be generated based on tracking position and/or motion of the input device without requiring contact with a surface."

Apple's patent FIGS. 10A-10B noted above illustrate an example configuration for generating content using an input device on a non-touch-sensitive surface according to examples of the disclosure. FIGS. 10A and 10B include different configuration of a two part input device #1000 such as a stylus #1002 and a cap #1004.

Patent FIG. 10A illustrates a configuration in which the stylus and cap are coupled together (e.g., similar to a conventional pen with a cap). FIG. 10B illustrates a configuration in which the stylus is decoupled from cap. The stylus can include a transmitter #1006 and the cap can include multiple receivers #1010 for receiving signals transmitted from the stylus. The transmitter and receivers can be used to track position. The cap can also include a memory, wireless communication circuitry and/or processor to store, transmit and/or process the data received by receivers. The stylus can be used to draw or write on a nearby surface (e.g., table, window, user's body).

Apple's patent FIG. 11 noted above illustrates another example configuration for generating content using an input device on a non-touch-sensitive surface. An input device, such as stylus #1102 (Apple Pencil), can include a transmitter #1103 which can operate like the transmitters. Receivers can be included in various devices to detect and track the stylus location.

For example, as illustrated in FIG. 11, the operating environment can include various devices such as desktop computer 1104, laptop computer 1108 and charger 1112. Desktop computer 1104 can include one or more receivers 1106, laptop computer 1108 can include one or more receivers 1110 and charger 1112 can include one or more receivers. The position and movement of the stylus can be tracked by one or more of the receivers in the various devices.

Generating 2D and 3D Content

Apple further details that in some examples, Apple Pencil (input device) will be able to generate two-dimensional content (2D) or three-dimensional (3D) content. Their patent FIG. 13 above illustrates an example of three-dimensional content created by an input device.

To generate a 3D shape, such as 3D cylinder #1300, Apple Pencil #1302 can trace one or more surfaces of a cylinder with one or more strokes. For example, a first stroke #1304 can trace a diameter of the top surface of cylinder and the height (along a length) of the cylinder. Second and third strokes #1306 and #1308 can trace the circumference of the cylinder.

The captured 3D strokes can be used to render a 3D shape in modeling programs such as computer-aided design (CAD) software or animation software. For example, as illustrated in FIG. 13, 3D cylinder #1320 can be rendered on the display of computer #1310 based on strokes #1304, #1306 and #1308.

Rendering 3D objects in this way can provide a simpler starting point for users than entering all of the dimensions manually into 3D modeling software. In some examples, the 3D tracing of objects can be used to generate models for 3D printers in a simple way.

In some examples, tracing or point samples can be recorded and used to accurately measure dimensions for content. The dimensions can be used to create an electronic canvas with appropriate dimensions or can be used to scale content to fit the measured dimensions. For example, picture frame dimensions can be measured using tracing or point samples, and the dimensions can be used to scale an existing photograph to fit the frame. In some examples, the dimensions can be used to create an electronic canvas of proper dimensions so that a user can create content that will fit in the frame dimensions without requiring scaling than may distort the content. In some examples, the user can provide a scaling parameter to scale the dimensions recorded by the stylus (before or after generating and displaying the content).

Augmented & Virtual Reality

Additionally, the input device and content generation described herein can be used in virtual reality or augmented reality contexts. For example, the input device (e.g., stylus or other sensor augmenting a finger or other body part) can be used to trace 2D or 3D physical surfaces or capture physical objects using point samples and the captured strokes or point samples can be used to render surfaces or objects in augmented or virtual reality displays as 2D or 3D surfaces or objects. In some examples, the captured physical content can be scaled before display in the virtual or augmented reality environment to match the relative size of objects in the virtual or augmented reality environment.

Additionally or alternatively, in some examples, the input device can be used to interact with objects displayed in the augmented or virtual reality environment. In some examples, the input device can be used as a selection or editing tool for interaction with GUI objects in a virtual or augmented reality display. For example, some or all of a computer file system can be displayed using augmented or virtual reality icons, and a user can tap or double tap (or selection means) with the input device at the location of the augmented or virtual reality icons to open or activate the icon. In some example, the input device can be used as an editing tool to modify the shape of the 3D cylinder of FIG. 13 (e.g., elongate the cylinder, change the radius of the cylinder, etc.) displayed in an augmented or virtual reality environment.

In some examples, the input device can be used to capture content displayed in virtual or augmented reality environments and the content can be displayed outside the virtual reality or augmented reality environment (or in a different virtual or augmented reality environment). For example, a 2D or 3D shape or object can be captured by tracing a virtual or augmented reality object, using point samples of the virtual or augmented reality object, or otherwise selecting the virtual or augmented reality object, and the captured shape or object can be displayed on a touch screen or other display device (or other virtual or augmented reality environment), thereby extracting the virtual or augmented reality object from that environment, and placing the content into a different environment.

Apple's patent application 20180018057 was filed back in Q3 2017 with some history dating back to Q3 2016. It was made public on January 18, 2018. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

One of the engineers listed on the filing is Tyler Bushnell who is a Product Designer at Apple. Coincidentally, Bushnell was also behind a patent that we covered in back in October 2017 in a report titled "Apple Wins Patent for Reinventing the Watch Band with Auto Adjustment and more."

Apple was granted a patent for that in October 2017 and last week Apple filed a "continuation patent" that updated their patent claims to cover a "bladder" to a future Apple Watch band that could be tightened and or eased to assist in adjusting the band for the user.

In Patent Claim #1 Apple notes: "A watch comprising: a housing; a processor within the housing; and a band coupled to the housing, the band configured to hold the housing against a wrist of a user; and an inflatable bladder configured to, in response to an electronic signal from the processor, adjust a tightness of the band on the wrist of the user."

Apple filed this as a continuation patent last week and it is not a new "granted patent" as Apple World Today is claiming today.

Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.