It's Time to Think Different, Again

Apple's inventor Brett Bilbrey provides us with an overview for this system: "Computer systems, including personal computers, are obviously well known. Conventional computer systems include basics such as a hard drive for storage, memory, video card and input devices like the Magic Mouse, Touchpad and Multi-Touch displays. In general, computers are capable of storing information or data from various input devices. However, the information collected by a computer from different input devices is typically unrelated and stored separately, other than audio and video data that may be simultaneously collected and stored together as a single audio-visual file.

One known system that is different from most computers is a computer system that includes a "smart pen" such as "Livescribe." The smart pen includes an infrared camera that captures everything that is written or drawn on a page of paper using the smart pen and a microphone that captures audio. The infrared camera and the microphone capture such visual and audio data while the smart pen is in a "record" mode, thereby associating the audio data with the video data captured at the same time. Once recording is stopped, audio that was recorded when a writing/audio writing/drawing was created may be played back by the smart pen by touching the paper at that writing/drawing with the smart pen."

On Livescribe's homepage there's a promotional video where you could hear about the value of the pen from the perspective of people in business and a student. The student states that "When I'm in class, it remembers everything I hear, write and draw." Which is exactly what Apple's engineer and inventor Bret Bilbrey is describing in this patent.
 

In further describing Livescribe's smart pen, Bilbrey states that the smart pen is capable of storing the recorded audio and visual data locally. Also, the smart pen is configured to transfer the data to a computer, such as a laptop or desktop computer, via a universal serial bus (USB) connection. The computer may store the visual data as page images with the associated audio linked to the particular writings/drawings. Once transferred, the data may be searched for keywords. The keywords may be identified in the stored page images, for example, by highlighting. Selecting a highlighted keyword may cause the associated audio data to be played back.

In Apple's Patent Point # 102, Bilbrey states that "it should be understood that various aspects described [in this patent] may enable unique implementations for data storage and retrieval not contemplated or possible with known computer systems. For example, notes to friends and/or the general public may be input (e.g., typed, written or spoken) or otherwise collected (e.g., recorded) and date and/or time stamped and/or location tagged. Others may then search for the particular date, time, person (e.g., author or speaker) and/or location to retrieve the note." The smart pen, according to this patent will work together with "mobile devices such as a smartphone or a tablet device" to retrieve various data described above."

Apple's patent describes many modules including face recognition, voice recognition and others that you'll see below in patent FIG. 4, but one of the key modules in context with a smart pen system is the "keyword module" which may be configured to process or analyze primary data, such as audio and/or video data, to identify one or more keywords that occur in the primary data. Any suitable techniques for recognizing spoken words, such as those employed for dictation software, language learning software, voice-activated devices, or the like, may be used. Also, any suitable techniques for recognizing written words may be used.

The patent also states that the system may also "be configured to identify authors of text and/or drawings that occur in video data of the primary data. For example, this may be performed through motion determination, such as movement of an authors hand and/or a writing instrument."

This is sophisticated technology as Bilbrey notes that "In addition to techniques employed by military and law enforcement, techniques employed in modern digital cameras may be used."

In Apple's Patent Point # 063, Bilbrey states that "It should be understood that the input/output interface may be any suitable device or combination of devices that is/are configured to receive user input and to provide a suitable form of output. Thus, any known device(s), such as a keyboard, a display or monitor, a touch screen, a mouse, a trackball, a stylus, a microphone, a speaker, or the like, or any device(s) hereafter developed may be employed.

It's About a Total Data Collection System

There's no doubt that a smart pen is likely to be one of the key input devices for this advanced data tagging and retrieval system for use with future mobile iOS devices. But it is one great tool in an overall master data tagging and retrieval system that Bilbrey describes in this patent.

To understand that the system is about a total data collection and retrieval system, Bilbrey notes that "for the sake of simplicity, it should be understood that other input devices, such as a digital camera, a cellular phone, a separate computer device (e.g., personal computer, laptop computer, handheld device, or the like), or other "smart" devices, may be employed. Such devices may be configured to perform some of the analysis of the data collected, as appropriate or desired."

Apple credits Brett Bilbrey as the sole inventor of this March 2011 patent application which was originally filed in Q3 2009. Just to ensure you that he's an Apple engineer, you'll note that Mr. Bilbrey recently worked on Apple's "Smart Bezels + Reconfigurable MacBooks" patent along with this week's patent about the Chromatic Magic Mouse. 

In recognition of Apple's ongoing work in this area, Patently Apple has opened a new Archive section to keep track of this ongoing trend under: Patents: Smart Pens.

Apple's iWeb is headed for the iPad

In another interesting Apple patent that recently surfaced, we learned about an electronic document authoring system under patent application 20110074710; a patent that lists Apple's UI Design Manager Christopher Weeldreyer as the lead inventor.

In that patent, Apple lists the apps that have been designed to work with gestures and more specifically touch screen displays like the iPad. The list includes productivity apps such as Keynote, Pages, Numbers and iWeb: backup – iWeb; Rewind – iWeb. Yes, the only one on this list that Apple has yet to introduce on the iPad is iWeb – and it's clearly marked in this patent as just a natural app for the iPad's electronic document authoring system. I'd say that pretty much confirms that it's on Apple's list of coming apps for the iPad sometime in the next year.

Notice: Patently Apple presents only a brief summary of patents with associated graphic(s) for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for further details. Patents shouldn't be digested as rumors or fast-tracked according to rumor time tables. Apple patents represent true research that could lead to future products and should be understood in that light. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.

Here are Some Great Community Sites Covering our Original Report

MacSurfer, Apple Investor News, Google Reader, UpgradeOS, iPhone World Canada, Apple Linkage Japan, TechWatching, PeepCode, CBS MarketWatch, 9 to 5 Mac, Macnews Germany, iGuides Russia, Apple+Z Germany, Electronista, MacKozer Poland, MobileMac Netherlands, MacWorld Sweden, iPhoneHellas Greece, iPadclub Netherlands, Macity Italy, Macnotes Germany, Macerkopf Germany, AllNewsMac, MacTechNews Germany, PadGadget, iPhone-MacNN, iGeneration France, Geeky Gadget, MacPlus France, i-ekb Russia, Melamorsicata Italy, Muj Mac Czech Republic, iPhones Russia, The Loop, Dobreprogramy Poland, Gadget Helpline, iCreate Magazine Netherlands, AppAdvice, MacObserver, iLounge, iPhonepyssel Sweden, Online PC Switzerland, MacWelt Germany, Jablickar Czech Republic, iPadCreative and more.

Note: The sites that are linked above offer Apple community members an avenue to make comments about this report in many original languages. Additionally, many of these sites provide our guests with different takes on any given patent or concept  presented in our reports to make it more fun, interesting and/or personal. If you have the time, join in!