Apple Wins an AR Patent that made their WWDC 2017 Demo of moving Virtual IKEA Furniture on a table Come to Life
Patently Apple posted a granted patent report back in April covering an Augmented Reality invention that they inherited when acquiring Germany's Metaio. The theme of the invention related to moving virtual furniture in the home or office before committing to its placement in real life. Today Apple was granted a second related patent from Metaio relating to two Augmented Reality applications. The first is for an "Interior Design Application" while the second is for a new "Movie Poster Application." Some of this invention may have ended up in Apple's new ARKit as the patent notes that it provides "tools for authoring."
If you saw Apple's first AR demo at this year's WWDC event then you know that when Apple's SVP of Software Craig Federighi moved virtual furniture around on a demo table, the crowd reacted with amazement at the accuracy of the virtual elements moving in realtime. This is one of the inventions that made that happen.
Yesterday Patently Apple posted a report titled "IKEA is excited about their new Augmented Reality App coming to iOS 11 with its Stunning Furniture Placement Accuracy." Of course being stunned by the accuracy due to ARKit means that the accuracy came from AR patents like the one presented today that describes an interior design application.
Granted Patent: Method of Image Processing for an Augmented Reality Application
Apple's newly granted patent covers their invention relating to a method of image processing in an augmented reality application, comprising the steps of providing at least one image of a real environment and performing image processing in an augmented reality application with the at least one image employing visualization of overlaying digital information with visual impressions or the image of the real environment and employing vision-based processing or tracking. The invention also relates to a computer program product comprising software code sections for performing the method.
Accordingly, the present invention is capable to improve the performance and usability of augmented reality applications, such as authoring tools. Particularly, the invention attends to the problem of user unsatisfied visualization of overlaying the virtual information or digital content to an image or view of a real environment in augmented reality or visual search applications that employ vision-based detection, localization or tracking solutions by adjusting parameters including their values and/or operating flow (workflow) of the vision-based processing or tracking according to the usage of the image processing or the visualization, according to a visually perceivable property of the digital information and/or the real environment, according to display devices, and/or according to the way of user viewing the visualization.
In Apple's patent FIG. 1 noted below we're able to see a flowchart of an exemplary augmented reality visualization approach according to the state of art. In step 1 an image of real environment is captured by a capturing device, particularly a camera. In step 2.a, vision-based processing and/or tracking is performed on the captured image. The virtual information provided in step 2.b is overlaid onto the image of the real environment in step 3. In another application, the virtual information provided in step 2.b may be overlaid onto the view of the real environment which is provided by an optical-see-through display device, as commonly known in the art.
The visualization of overlaying digital (e.g., virtual) information onto the image of a real environment could be used for different purpose. Augmented reality visualization is often used to enhance user's visual perception by superimposing virtual information or digital content to the real environment in a correct perspective.
For example, augmented interior design could superimpose a virtual sofa to the camera image of the real environment of the room in a correct perspective. This application could visually support furniture arrangement in the room without having to move them physically. For this, an accurate pose of the camera with respect to the room is mandatory.
Apple's patent FIG. 4 noted above illustrates a flowchart of a method according to an embodiment of the invention overlaying a virtual object (here a furniture, like a sofa) onto the image of the real environment in an augmented interior design application based on an embodiment of the present invention demonstrated in FIG. 2. The property of the virtual information, such as symmetry, behavior, color, etc., and/or its usage is checked in step 2.c, wherein the method in response to this information configures or adjusts the processing or tracking method accordingly. For example, in step 2.d the threshold value in the similarity measures of sum-of-squared differences is increased. In step 3, the virtual sofa is then overlaid onto the image of the real environment.
In Apple's patent FIG. 5 noted above we're able to see a flowchart according to an embodiment playing a video (as the digital information to be overlaid, step 2.b) in full screen mode in an augmented movie poster application based on the embodiment of the present invention demonstrated in FIG. 2. As a result of step 2.c, as described above, the threshold value in the similarity measures of sum-of-squared differences could be decreased. In step 3, the video is played in full screen mode.
Many vision-based tracking solutions cannot meet both of accurate pose estimation and high recognition rate. For example, a low threshold value in the similarity measures of sum-of-squared differences could result in an accurate estimated pose, but may lead to a low recognition rate. On the other side, a high threshold value for the similarity measures could result in an inaccurate estimated pose, but will lead to a high recognition rate. Therefore, adjusting the value of the threshold depending on the usage of the augmented reality visualization could lead to a user satisfying visualization.
The visual properties of the virtual information or the real environment are, but not limited to, the spatial property, behavior, and color.
When virtual information is a spherical object with homogenous texture on its surface, rotating the spherical object around the spherical center will not result in different visual perception to users. Translating the spherical object will result in different visual perception to users. Therefore, in the optimization process, parameters describing the translation of the virtual object have to be optimized, while parameters describing the rotations around the spherical do not need to be considered.
In Apple's patent FIG. 3 noted above we're able to see a flowchart according to an embodiment overlaying a virtual spherical object as the digital information onto the image of the real environment based on the embodiment of the present invention demonstrated in FIG. 2. In step 2.b, a virtual spherical object with homogenous texture on its surface is provided. As a result of step 2.c, as described before, parameters describing the rotations around the spherical center are adjusted in a way that they are not considered. In step 3, the virtual spherical object is overlaid onto the image of the real environment.
Apple's granted patent 9,685,004 was originally filed in Q2 2014 and published today by the US Patent and Trademark Office.
Patently Apple presents only a brief summary of granted patents with associated graphics for journalistic news purposes as each Granted Patent is revealed by the U.S. Patent & Trademark Office. Readers are cautioned that the full text of any Granted Patent should be read in its entirety for full details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.