Last Week the US Patent & Trademark Office published a patent application from Apple that could be signaling a new feature and app coming to Apple Pencil that will allow users to create 3D images on their iPad that could be used for creating art or logos, very much like Microsoft's Paint 3D app. In the bigger picture, Apple envisions the use of this technology to also be used in conjunction with a future mixed reality headset, something we've covered extensively over the last few days. (see one, two and three).

Microsoft introduced their new 3D modeling app for mere mortal last year with Windows 10 Creators update. Paint 3D incorporates features of Microsoft Paint and 3D builder to combine a lightweight hybrid 2D-3D editing experience that allows users to pull in a variety of shapes from the app, their personal computer, and Microsoft's Remix 3D service. The video below provides you with a quick overview of Paint 3D.  

Apple's patent FIG. 3 presented below illustrates a perspective view of a system including a stylus for measuring coordinates of a surface supporting an object, according to some embodiments of the subject technology; FIG. 4 illustrates a perspective view of the system of FIG. 3, including the stylus for measuring coordinates of the object on the surface.

According to Apple, the stylus (Apple Pencil) # 100 can be used to virtually draw in three-dimensional space. Illustrations can be created in free space by motion of the Apple Pencil by the user.

One or more aspects of calibration and motion tracking for Apple Pencil can be applied to track free motion during a drawing session. The user can operate a new button #150 on this future version of the Apple Pencil (illustrated in patent FIG. 2 further below) to initiate a drawing session, in which a marking is virtually created starting at the location of the tip probe #190 in the coordinate system.

As the user moves their Apple Pencil and continues the drawing session, markings continue to be generated along the pathway of the tip probe. When the user provides an indication (e.g., by releasing or pressing the button 150) to cease the drawing session, the marking ceases to be generated, and the user can move the Apple Pencil without generating additional markings.

The recording of markings can be performed independently of whether the tip probe makes contact with a surface. According to some embodiments, the Apple Pencil will be able to record motions by the user during a drawing session and transmit the applicable data for later display.

According to some embodiments, the Apple Pencil is able to communicate with a device for visual display to the user, with which the markings are displayed for the user as they are generated during a drawing session. For example, the display can include a virtual reality headset in communication with Apple Pencil (e.g., directly or via an intervening device). By further example, the display can include a flat-panel display that displays a three-dimensional rendering of the markings.

During or between drawings sessions, the button #150 of the Apple Pencil or another input device can be used to change how the markings are generated. While the Apple Pencil is used for drawing or writing, the positions of the tip probe can be recorded as markings, lines, or shapes having a variety of characteristics.

For example, the recorded markings can have a certain shape, thickness, and color. When the user operates the button or another input device, the Apple Pencil or another device can interpret the signal as a command to apply one or more characteristics to markings generated by subsequent input from Apple Pencil. Accordingly, subsequent markings can be recorded as having the one or more characteristics determined by the input signal.

According to some embodiments, a drawing session can be used to modify or edit a model generated based on operation of Apple Pencil with an object. For example, a user can measure an object, command the object to be rendered and displayed, and then further operate Apple Pencil to change aspects of the rendered and displayed model.

Further Apple notes that Apple Pencil will be able to map motions in three-dimensional space for use in various other applications. According to some embodiments, the motion of Apple Pencil by the user can be used by a gaming application that uses motion as an input.

In some embodiments, the motion of Apple Pencil by the user can be used to detect user gestures that indicate user commands to an external device for performing a particular action. According to some embodiments, the motion of Apple Pencil by the user can be used to detect a selection by the user of an object that is displayed to the user in a virtual three-dimensional environment.

Apple's patent FIG. 8 presented above illustrates various components of a possible future Apple Pencil described in this patent application

According to some embodiments, the Apple Pencil could include a probe sensor #192 at a tip probe #190 of the Apple Pencil for sensing when the tip probe is contacting a surface, such as the surface #50 or the object #20. The probe sensor can include one or more contact sensors, capacitive sensors, touch sensors, cameras, piezoelectric sensors, pressure sensors, photodiodes, and/or other sensors. The probe sensor can include a capability to sense distortions on the surface, such as by detecting varying forces with a force sensor. Distortion data can be collected to be included in topographical information regarding surface features of an object being measured by Apple Pencil.

Apple's patent application #20180081455 was filed back in Q3 2017 and published last Thursday by USPTO.

One of the inventors of this invention is Product Designer Joel Ruscher who has been with Apple since 2013. Some of his specialties include - Conceptual Space Development/Ideation; Architectural/Platform Development; R&D for Consumer Electronics, Defense, Medical; 3D/CAD Specialties and more. The other inventor is noted as being Product Designer Kevin Armendariz.

Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.