Apple's Patent Background and Overview

When a user interacts with others in various communication contexts (e.g., in online chat sessions, emails, etc), the user could sometimes enter textual strings or preset emotional icons ("emoticons") in a text message to reflect his or her current emotional state (e.g., happy, angry, sad, etc.) to other users. These emoticons help improve the interpretation of the plain text of the message.

Apple's Solution

Apple's patent states that conventional avatars and emoticons are often static and generic. Even though users are allowed to customize the appearances of their avatars to a certain degree, the users have to rely on preset emoticons or canned avatar faces other than their own individualized avatars to express the emotional connotations in their text messages and communications. The subtle differences in how each individual expresses his or her emotions could not be captured by these preset emoticons and canned avatar faces.

By allowing individual users to associate individualized avatar expressions and/or body language with particular user states, the users are given the means to communicate with others in a more accurate and expressive way. When users are allowed to create and use avatar instances having individualized expressions and/or body language for each user state, the users could enjoy more creativity and fun in the self-expression and the interactions with other users.

The Patent's Focus: Replacing Emoticons with Avatars for Texting & Email

Apple's patent FIG. 1 shown below illustrates what Apple deems to be "an exemplary avatar operating environment" where users could create and customize their individualized avatars to represent themselves in various virtual environments. Apple states that in some implementations, the avatar server 102 could provide an avatar editing environment in which each user creates his or her own individualized avatar. The extensive detailing of this "editing environment" was first presented by Patently Apple last month. .

User States

Yet today's patent isn't really about the editing environment as much as it is about illustrating that users will be able to define different user states and create different instances of their individual avatars to express their emotion associated with each user state.

As an example, the definition of a user state could include a name or identifier for the user state. A keyword, such as "Happiness," "Surprise," "Joking," and so on, could be used as the name or identifier of a user state that is associated with the emotion indicated by the keyword. The name or identifier could also be an emoticon entered by a user.

Trigger Events & Avatar Instances

The definition of the user state could further include the trigger events that indicate the presence of the user state. The trigger events could be a particular keyword such as "Wow." Once the trigger events for a user state are specified, the user could create and associate a particular avatar instance with the user state. The avatar instance created by the user would have the facial expressions, body language, and/or presentation theme, that the user wishes to use to represent the emotion that the user is likely to have upon occurrence of those trigger events. Apple's patent FIG. 2 below illustrates a flow diagram of an exemplary process for using customized avatar instances to reflect user states.

For example, if the trigger event is a keyword "WOW," the user may create an avatar instance based on his own individualized avatar that has a facial expression of amazement and excitation in a degree that is in line with how the user himself may express this emotion.

In some implementations, a default set of user states could be provided by the avatar server 102 to each user. For example, some basic emotions and associated trigger events (e.g., common emoticons) could be provided in the default set, and each user could add or delete the user states as the user wishes.

A default set of avatar instances could also be provided for each preset user state. Each avatar instance could be based on a modification of the basic individualized avatar of the user using heuristics derived from people's facial expressions for different emotions. The default set of avatar instances could be fine-tuned by each user so that they more accurately reflect the subtle differences in how each user would express the same emotion.

Modifying the User State

In Apple's patent FIG. 6A below we see example interfaces for defining and modifying a user state. The illustration on the right shows us that the user will be able to "Add a State" as well as "Edit a State. The user interface is also divided in a simple-to-use form which connects a "state" with a "trigger" that equates to a particular "Avatar" that will replace the traditional emoticons the user previously used. For instance: The user will type in a text message or email with the usual Happy State represented as ;-). Instead of getting a traditional round white or yellow static smiley face emoticon, the new system will automatically equate ;-) to that of a custom Avatar that you designed yourself with the use of the editor environment that we presented last month.

The illustration on the left shows us that the user will be able to edit the name or identifier of the user state in field 624. The user will select and type in a new name for the selected user state "Happy," such as "Smile." As you could see in the first column of the left side illustration, that there are a number of states that you'll be able to control. Here the illustration has a state for "Surprised, ""Birthday" and "Focused." Each will have a card on file that matches the face of your choosing to that state. Apple states that the types of trigger events are only limited by the user's creativity and imagination.

Creating a Customized "Avatar Instance" for a Selected "User State"

Apple's patent FIG. 6B shown below illustrate a few example user interfaces 600c for creating a customized avatar instance for a selected user state.

Customizing Features

One of the key features of this user interface is the element picker noted as patent point 648 which allows a user to add or modify an element of the avatar instance. When the user selects this feature, they'll be able to select a particular element (e.g., face, skin tone, eyes, nose, hair, shirt, hat, background, etc.) of the avatar instance to edit the element in detail.

Another key feature is the adjusting slider noted above as patent point 654.With this feature, the user will be able to change the intensity level of the emotion of the avatar instance. For example, when the user will move the indicator 656 toward the right extreme of the slider, the emotion on the avatar face becomes more exaggerated (e.g., with enlarged eyes, raised brows, and wide open mouth). In contrast, when the user will move the indicator toward the left extreme of the slider, the motion on the avatar face becomes more subdued.

The last feature to point out is that of the element picker noted above as patent point 674. The element picker is about adding and modifying accessories, clothing items, and presentation themes associated with an avatar instance.

Option: Define a "User State" with a Photo

Apple states that in some implementations, the user state could be defined by a photograph of the user showing a particular emotion. For example, the user could upload a photograph showing a big smile to the avatar server, and use the photograph as a trigger event for a "Happy" user state. So for adults, this option could be more appealing. To add a little fun, Apple hints that the user will be able to modify their appearance in Photo Booth and tag that as a particular user state (Happy, Hung Over, Angry, Crazy etc.). Either that or the imaging editor will be very Photo Booth-like. It makes sense considering that you could "Create a Buddy Icon with Photo Booth," today.

I think we've all played around with Photo Booth a bit at some fleeting moment for laughs. Using it as an editing tool for Apple's Avatar app is very interesting and potentially very entertaining when creating your own set of photo-based emoticons. It's only good business to repurpose the Photo Booth engine for another application.

An image of the Photo Booth-like interface could be found in Apple's second avatar centric patent this week found under number 20110292051.

The image found in the secondary avatar patent happens to match up with a user state defined in the main patent. Apple's patent states that the user will be able to define an "Annoyed" user state having a trigger event based on an undesirable device state. Uniquely Apple states that you'll be able to set an "Annoyed" user state to be associated with such things as a "low battery level" below 20 percent or your "network bandwidth" being below a certain threshold.

Yet I could see how your supervisor at work will abuse this feature in no time flat. That's all you need now, right? Sure, a photo personalized avatar on emails from your supervisor with facial expressions of "annoyance" and/or "Anger" over something that you just didn't do right according his pernickety standards: Oy.

Apple Working on Delivering Highly Advanced Photo-like Avatar Tools

In Apple's secondary Avatar-centric patent application, they focus on higher end 3D Avatars. Apple states that a three-dimensional ("3D") avatar could be automatically created that resembles the physical appearance of an individual captured in one or more input images or video frames. The avatar could be further customized by the individual in an editing environment and used in various applications, including but not limited to social networking, video conferencing and gaming.

Apple further states that a genetic process could be used to efficiently estimate key parameters describing an avatar that resembles the person in the input image. During the genetic process, an individual "genome" could be evaluated by generating an avatar according to parameter values in the individual genome. In some implementations, the individual genome parameterizes a 3D avatar model, which could be rendered on a device using a graphics engine.  

The 3D model could include an avatar head with adjustable widths at multiple elevations, as well as several mesh models for hair, nose, eyes, mouth, ears, glasses, mustache, etc. Each of these mesh models could be scaled, positioned, and oriented under parametric control.

In January 2011, Intel's Mooly Eden pointed to new ways that we'll be communicating with others in the future that heavily used high-end 3D Avatars. Eden pointed out how these high-end avatars would be used in texting, email and allow us to one day transpose our avatar image into gaming characters found in our favorite games (like Call of Duty or Assassin's Creed).

So make no mistake about it, avatars won't be limited to funky cartoons such as those presented in patent figure 6B above. They could be very realistic as we see with Mooly Eden's image.

Looking way out in the future, imagine when we'll be able to send a next generation email that sends your 3D Avatar to your friend and it vocalizes your message and with the assistance of limited version of Siri's engine, be able to take back a voice message from your friend. Hmm I don't know what I'd think of that. Would that be freaky, fun or terrifying? – Ha! Time will tell.

Then again, quoting Intel's Mooly Eden: "In three, four years, the way that we're going to communicate with our devices is going to be totally different. When we look at the keyboard and the screen, the way they are today, they'll look like the middle ages." Obviously Eden is seeing the research in their labs and it must be really "out there" if he's making that prediction.

And lastly, it should be noted that while Apple mentions gaming this passing, they do so in passing. The true focus of this main avatar patent is on being able to use personalized avatars as traditional "emoticon replacements" in texting and email – just as their patent background confirms.

Apple's initial avatar-centric patent application covered in this report was originally filed in Q3 2010 by inventors Thomas Goossens of France and two Californians by the name of Laurent Baumann and Geoff Stahl.  

Report Out Take: Hmm, is it just me, or does that term "genetic process" sound a little too much like something right out of Dr. Frankenstein's playbook? Ha! Apple, that's one creepy term that you should seriously consider renaming.    

Sites that are following our Original Report: MacSurfer, Huffington Post Canada, Twitter, Facebook, Apple Investor News, Google Reader, Macnews, iPhone World Canada, MarketWatch, SlideToMac Italy, AppleZein Italy, and more.