Apple has been working on advancing a future iPad with various forms of advanced haptics for some time now. Back in 2012 we covered Apple's most advanced haptics invention to date in a report titled "Apple Looking to deliver a little More Buzz to iOS Devices." At the time there was buzzing rumors of Apple adopting technologies from Senseg which never materialized but the rumors were on target that Apple was working on sensing technologies that they could one day bring to the iPad. Today the U.S. Patent & Trademark Office published one of Apple's most advanced inventions for a future stylus that could sense textures and far beyond. Apple has been working on a future stylus or iPen for some time now and you could check out our archives that illustrate a wide variety of ideas that Apple has on this future device. Today's report covers Apple's invention that involves a stylus with texture capture, advanced haptic feedback and more importantly an inertial measurement unit. This type of technology is currently used in many navigation systems such as for aircraft, watercraft and other forms of transportation. The data collected from the IMU's sensors allows a computer to track the position using a method known in the art as dead reckoning. All in all, I'd have to say that this is one of Apple's top stylus inventions to date.

Apple's Patent Background

In the visual arts, texture is the perceived surface quality of an image or work of art. It is an element of two-dimensional and three-dimensional design and is distinguished by its perceived visual and physical properties. Use of texture, along with other elements of design, can convey a variety of messages and emotions. For example, use of rough surfaces can be visually active, while smooth surfaces can be visually restful. Physical texture, also known as actual texture or tactile texture, are the actual variations upon a surface. Physical texture has a physical quality that is a combination of how the surface looks, and how it feels to the touch. For example, in a painting, physical texture is based on the paint, and its application, or the addition of materials such as ribbon, metal, wood, lace, leather and sand either on the surface of, or mixed with, the paint.

Visual texture is the illusion of having physical texture. Every material and every support surface has its own physical texture that needs to be taken into consideration before creating a visual image which is meant to emulate that physical texture. For example, materials such as canvas and watercolor paper are considerably rougher than, for example, photo-quality computer paper and it may be difficult to create a flat, smooth texture from an image shown on such materials. Photography, drawings and paintings use visual texture both to portray their subject matter realistically and with interpretation. Visual texture in those media is generally created by the repetition of shapes and lines. Simulated texture creates the visual effect of texture without actually adding texture. For instance, texture may be created to look like something other than paint on a flat surface. Alternatively, an image may create the illusion of ripples through the repetition of lines.

Today, the term stylus generally refers to an input tool used with Personal Digital Assistants (PDAs), graphics tablets, Tablet Personal Computers (PCs), and Ultra Mobile Personal Computers (UMPCs). Modern day styli generally take the shape of a writing instrument, such as a pen, and are made to comfortably fit in the grip of a user's hand. These styli can be found in all different sizes and shapes. Some styli may extend and contract into small, pen-like cylinders, which are easy to store. Other types of styli may be larger or contain additional devices such as microphones.

In electronic applications, the user operates a touchscreen with a stylus, rather than using a finger in order to avoid getting the natural oil from the user's hands on the screen. Use of a stylus also improves the precision of the touch input, allowing use of smaller user interface elements such as icons. Styli may also be used for handwriting or drawing on a touch-sensitive surface such as a computer tablet screen and may assist a user to accurately navigate through menus, send messages etc. As electronic devices have evolved, they have been used more frequently by engineers, artists and other creative individuals for artistic drawings, image creation, graphic design and other uses.

When a user of an electronic device, such as a graphics tablet or a PC, attempts to create an image which includes visual texture, that user may be limited in the type of physical texture he or she can simulate. Thus, it is generally left to the creativeness and efforts of the user to determine the methods and types of visual texture which may be provided. Accordingly, an improved device, method and/or system for capturing, storing, and providing visual texturing information from an image of a sensed surface to an electronic device may be useful in permitting the user to create an image which emulates the texture of the surface that was sensed.

Apple Invention Covers a Stylus with Texture Capture

Apple's invention generally relates to styli that may include additional components for sensing, recording and transmitting an image of the texture of a surface.

The stylus may include an image sensor contained within or attached to the stylus which senses the physical attributes of a surface over which the stylus is passed. The stylus may include a light transmissive tip through which the image sensor can detect the texture of the surface over which the stylus is passed. The sensor may be a camera which may be contained within the stylus or may otherwise be associated with the stylus. The image which is sensed through the light transmissive tip of the stylus may be stored or immediately utilized to produce an image which visually simulates the texture of the sensed surface or image.

Apple's patent FIG. 4 noted above illustrates a stylus #14 which includes a light transmissive tip #15, and an off/on button #16. According to Apple, the stylus tip is configured to act as a lens for an image capture device such as a camera #17 or other suitable image capture device. The camera may be a fine scale camera or other suitable imaging device which is used to sense the depth of the image over which the tip is passed. A pinhole camera with minimal lens requirement may be used as camera.

It should be appreciated that certain embodiments may include a capture mode during which such variations, textures, features and physical characteristics, and in some cases colors, may be sampled or otherwise captured. Further, embodiments may employ an output or drawing mode in which the captured physical characteristics are reproduced on a display of an associated electronic device in response to motion of the stylus or other input from the stylus.

In Apple's patent FIG. 9 noted above we're able to see that the stylus includes an inertial measurement unit IMU #32 in addition to camera #17, power source #18, data transmission device #19, light source #21, and memory device #22. Including an IMU allows the stylus to be used to generate three dimensional images.

An inertial measurement unit (IMU) is used in many navigation systems such as for aircraft, watercraft and others form of transportation. The data collected from the IMU's sensors allows a computer to track the position using a method known in the art as dead reckoning. IMU's detect a current rate of acceleration using one or more accelerometers and detect changes in rotational attributes like pitch, roll and yaw using one or more gyroscopes. Some IMUs include a magnetometer to assist in calibration against orientation drift.

Haptic Feedback

In Apple's patent FIG. 11 we're able to see a haptic feedback device which may be, for example, a rumble pack #29 such as is known in the electronic gaming arts could be used in conjunction with a stylus to generate haptic feedback to a user.

An image, such as texture image associated with cloth #23 is captured by the camera stylus. That captured image is displayed on the screen of a tablet. The stylus is held by the user in one hand while the rumble pack is simultaneously held in another hand. As the stylus is passed over the tablet's screen the image is sensed by the stylus and the corresponding texture information is conveyed to rumble pack by software associated with the tablet. The user is thus able to visually observe the image on the tablet's screen while feeling the texture of the displayed image through rumble pack. The combination of visual and tactile input to the user allows them to experience the "look and feel" of a fabric.

Visually Impaired Application: In an alternate embodiment, a visually impaired person could use stylus 14 and rumble pack 29 as described above. That embodiment would allow the visually impaired person to "feel" an image which is displayed on screen 13 that he or she is otherwise unable to see due to the impairment.

Medical Application: The combination of two senses, visual and touch, makes this embodiment useful in other applications such as medicine where a medical professional may use this application to "feel" the image of an x-ray or a magnetic resonance image (MRI).

Sensing Surface Textures

Apple's patent FIG. 10 is a perspective view of a three dimensional map including surface features and a stylus. In FIG. 10, the stylus may be passed over three dimensional map #27 and images of surface variations or features, such as mountain ranges #28 could be captured and sent to the tablet 11 in order to create a visual two dimensional image of the captured three dimensional image #27 thus providing topographic detail to the two dimensional image.

In Apple's patent FIG. 7 it's noted that as the stylus is moved to generate portions of the visual image, the stylus may also provide haptic feedback corresponding to the texture, features, surface characteristics, and/or other physical characteristics of the stylus output, or an image generated in response to stylus output.

The ability for a user to sense surface texture variations adds a depth dimension to the visual images which may be generated. No longer will a user be limited by preprogrammed textures or surface features. A user can generate his or her personalized surface textures. These textures may then be integrated or combined in order to produce additional textures in generated visual images. Faux painting or faux finishing are terms used to describe a decorative paint finishes that replicate the appearance of materials such as marble, wood or stone. Using the stylus in these applications allows a user to generate realistic two dimensional images by using stylus to sense the texture of the actual surfaces. By employing haptic feedback, designers and modelers may receive touch feedback relating to the "surface" they are sculpting or creating, allowing faster and more natural workflow than traditional methods.

Another embodiment of the invention can be implemented to engrave jewelry or other items such as metals, plastics, glass and crystal. It is now commonplace for retail stores (mostly jewelry, silverware or award stores) to have a small computer controlled engraving capability on site. In Apple's patent FIG. 8 we're able to see an engraved necklace #25. Engraving #26 may take any decorative or other aesthetically pleasing form as is known in the art. The engraving could take the form of an inscription conveying sentiments or other information. Retail engraving machines tend to be focused around ease of use for the operator and the ability to engrave a wide variety of items including flat metal plates, jewelry of different shapes and sizes, as well as cylindrical items such as mugs and tankards. With that in mind, the user would be able to pass the stylus over a surface such as the engraving shown in FIG. 8 and the sensed texture could then be used by the engraving machine to reproduce that sensed textured surface on another item. In this way, copies of engravings, signatures etc. could be reproduced or custom engravings etc. could be produced by combining various sensed texture surfaces.

CAD Application

Another embodiment of the device may be used to create three dimensional images. In this embodiment the texture surface may be a three dimensional representation of a sculpture or other physically represented surface such as a tool design. In this embodiment, the stylus may be used as part of a three dimensional computer aided design (CAD) system. For example, the stylus may be used to design a complex surface required for a specific tooling application.

CAD uses computer systems to assist in the creation, modification, analysis, or optimization of a design. CAD software is used to increase the productivity of the designer, improve the quality of design, improve communications through documentation, and to create a database for manufacturing. CAD output may be in the form of electronic files for print, machining, or other manufacturing operations. CAD is also widely used to produce computer animation for special effects in movies, advertising and technical manuals, often called digital content creation. In one embodiment, texture images captured by the stylus may be utilized in digital content creation. That is, texture images stored in memory may be sent by transmission device 19 to a computer aided design system which may then implement the texture images in a digital content creation application.

CAD may be used in designing electronic systems, mechanical design or computer-aided drafting which includes the process of creating a technical drawing with the use of computer software. CAD software for mechanical design uses either vector-based graphics to depict the objects of traditional drafting, or may also produce raster graphics showing the overall appearance of designed objects. CAD may be used to design curves and figures in two-dimensional (2D) space; or curves, surfaces, and solids in three-dimensional (3D) space. The stylus could be used in the implementation of such design, particularly in the three dimensional design applications.

Patent Credits

Apple credits Jason Lor (who is currently on the Apple Team), Patrick Carroll, Glen Rhodes and Dustin Verhoeve as the inventors of patent application 20150212602 which was originally filed in Q1 2014. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Comments are reviewed daily from 5am to 6pm MST and sporadically over the weekend.