Today, the U.S. Patent and Trademark Office officially granted Apple a patent that relates to a possible future Vision Pro app that provides artists with a virtual canvas & sketchbook that will provide virtual erasing and smudging techniques and the ability to paint or draw in layers that could be edited.

Apple's newly granted patent describes techniques for providing an interactive computer-generated reality (CGR) environment for creating a virtual drawing on a drawing surface using an electronic system and device (such as Apple Vision Pro), where the CGR environment provides a user with a realistic and immersive experience while creating the virtual drawing as if the user was creating a real drawing in the real world. In this way, the described techniques enable a user to create a drawing without the need for traditional physical drawing tools.

Further, because the drawing is created within a CGR environment, the described techniques allow a user to easily and efficiently manipulate (e.g., change, erase, alter) a portion of a drawing as desired. As such, the described techniques enable a user to experience creating a virtual drawing within a CGR environment as if the user were creating a real drawing in the real environment, while simultaneously reducing some inconveniences associated with creating a real drawing in the real environment.

In accordance with some embodiments, a method is described. The method comprises: detecting, using one or more image sensors, a first input including a first predefined finger orientation type; and upon detecting the first input, drawing on the drawing surface within the computer-generated reality environment responsive to hand movement detected using the one or more image sensors, wherein drawing on the drawing surface within the computer-generated reality environment responsive to the hand movement comprises producing a visual effect on the drawing surface within the computer-generated reality environment corresponding to the detected hand movement.

Apple's patent FIG. 2 below illustrates the blank starting canvas for a virtual artists app designed for Apple Vision Pro; In FIG. 4 we see that the user initiates a gesture that will call up tools to use such as pencils or brushes; In FIG. 5 we see the user has chosen a virtual pencil for a sketch; In FIG. 6, the user begins their drawing; In FIG. 7 Apple notes that in some embodiments, when the user provides an input on the drawing surface using the drawing tool, device #100a generates (and/or causes a different device, such as a headphone, to generate) one or more (visual and/or audio) corresponding feedbacks associated with the drawing.

Further to FIG. 7, Apple notes that in some embodiments, the feedback includes an audio feedback (in addition to or alternatively to a visual feedback) corresponding to movements of drawing tool #206 on drawing surface #202. In some embodiments, if the selected drawing tool is a pencil, the audio feedback is a sketching (or scribbling) sound of a pencil being used on paper. In some embodiments, if the selected drawing tool is a painting brush, the audio feedback is a brushing sound of a brush being used on a canvas. In some embodiments, if the selected drawing tool is a pen, the audio feedback is a scribbling sound of a pen being used on paper.

In some embodiments, the magnitude of the sound feedback corresponds to the amount of force being applied to the drawing surface by the drawing tool. Thus, in some embodiments, the magnitude of the audio feedback increases or decreases in accordance with increases and decreases, respectively, in the amount of force being applied to the drawing surface by the drawing tool within the CGR environment.

In Apple's FIG. 9 we see the artist uses a new "open-palm" gesture that could initiate an artist's palette to work from with various paints available to use. Other gestures will allow the artist to apply a bold or light pen or paint brush stroke; FIG. 12 a user has initiated an "erase" mode to correct an error or a stroke they didn't like.

In Apple's patent FIG. 13 above, the artist uses a two-finger gesture for a "smudging" mode; FIG. 14, illustrates device displaying (or causing display of) CGR environment drawing surface #202 rotated to a side-angle view (from the perspective of the user). In some embodiments, drawing #208 comprises a plurality of drawing layers (e.g., a painting drawn using at least a first color and a second color, where the first color is applied over the second color on portions of the painting), thus providing a depth effect to the drawing, as shown in FIG. 14. The layers are editable. Apple describes it this way: because the drawing is created within a CGR environment, the described techniques allow a user to easily and efficiently manipulate (e.g., change, erase, alter) a portion of a drawing as desired."

The lead on Apple's patent is Edwin Iskandar, Senior Manager, visionOS.