Apple Won 2 Mixed Reality Headset Patents yesterday with one that could make Product Images Leap off a Printed Page and be Interactable
Apple fans know that one of the most important product lines ahead is their mixed reality headset family. Back in April 2021: During an interview with New York Times journalist Kara Swisher Apple's CEO Tim Cook agreed with her that augmented reality is "a critically important part of Apple’s future." He imagines AR being used in health, education, retail, and gaming."
One of the two granted patents issued to Apple yesterday that relate to HMD systems could be considered revolutionary once you wrap your head around the concept. We've all heard someone describing something that they've read in a book and described it as something that leapt off the page. Well, Apple's AR Team have taken that concept literally by allowing static images in a magazine or book quickly turn into a 3D virtual object that users could interact with on various levels.
Apple's granted patent covers various embodiments of electronic systems and techniques for using such systems in relation to various computer-generated reality technologies, including virtual reality and mixed reality (which incorporates sensory inputs from a physical environment). In particular, the patent provides techniques for displaying a virtual object corresponding to an electronic product in an AR environment. The electronic product is identified in physical printed media (e.g., magazine, newspaper, flyer, billboard, poster, and so forth), and then a virtual object corresponding to the electronic product is displayed in the AR environment.
The virtual object includes one or more software features (e.g., boot-up sequence, user interfaces) of the physical electronic product. User interactions with the virtual object cause the virtual object to be modified in a manner resembling the physical electronic product. For example, a user may interact with a user interface displayed on virtual mobile phone in a similar manner as would be done on a real-world user interface of a physical mobile phone.
Apple's 1B below illustrates System #100 which includes a base station #100b (Mac, iPhone or iPad) and a Head Mounted Device (HMD) 100c; FIG. 2 is an image of printed media #200 (a book, magazine, poster etc.); FIG. 3 is an iPhone/iPad with a picture of the book shown on its display.
In patent FIG. 3 above when an iPhone/iPad detects that the captured images include a depiction of electronic product such as a smartphone (204a), an affordance/icon is displayed to indicate that the depiction of the smartphone has been detected and that a virtual 3-D virtual object corresponding to the smartphone is available for display once the user taps on the display of the smartphone image, shown at #306 of FIG. 3.
Apple then notes that their future mixed reality Head Mounted Device (HMD) will detect the user touching the smartphone image on the display by following the user's finger using eye-tracking. Once the HMD acknowledges the touch has occurred, the HMD will replace the photo of the smartphone on the iPhone or iPad display with a virtual representation of the smartphone.
Alternatively, when the iPhone/iPad takes a photo of an advertised product like a smartphone image in a printed magazine, it may also detect a QR code in order to determine which smartphone is to be converted into a 3D object for the HMD viewer.
Possibly Interacting with 3D Object Software
Apple takes their concept one step further. They could make it possible for a consumer to actually interact with the software associated with the 3D object. Let's say the image and 3D AR object is displaying iPhone 20 with iOS 24. The user will be able to touch the virtual 3D image of the iPhone display and actually interact with iOS 24 as if they were at a brick-and-mortar Apple Store to see how features work etc. Now how insanely cool would that be.
The HMD will allow the user to then interact with the 3D object of the advertised item in a 360° manner. The image may also allow a user to interact with the software of operating system of the 3D version of the item such a smartphone (an iPhone, maybe Android phone etc.).
To be clear, Apple states: "In cases where the electronic product 204a is a depiction of a physical electronic product that includes a software feature (e.g., a user interface), the displayed virtual electronic product #404a includes an emulation #410a of the software feature. The emulated software feature #410a recreates the appearance and operation of the physical electronic product's software feature. In some embodiments, the emulated software feature #410a recreates the appearance of the physical electronic product's boot-up sequence. In some embodiments, the emulated software feature #410a is a predefined video depicting the appearance and operation of the physical electronic product's software feature."
With the HMD on, eye-tracking will be able to follow the user pointing to parts of the software to interact with one a menu as presented in patent FIG. 5 above. The menu shows an example of History/Company/Model/Year/Features. Of course, the menu would be different depending on the product and what the advertiser wants to emphasize about their product.
Obviously, Apple will have to work with their developers closely to pull this off, as the QR codes will likely have to be specialized for Mixed Reality Content. But knowing Apple, they'll make this an easy process. Advertisers should love this concept and who knows how far Apple could take this concept. For those not close to an Apple Store, Apple will make it easy for HMD users to interact with new devices and iOS specialty features and more as if you were actually in the store.
Apple's granted patent titled "An Electronic Product in an Augmented Reality Environment" credits Justin Stoyles (Engineering Program Management Lead AR/VR Applications & Content) and Michael Kuhn (Software Development Manager who came to Apple via their acquisition of Metaio GmbH in 2015). The patent shows that work on this began in 2017, two years after the Metaio acquisition, so it's definitely an Apple original concept for their future HMD.
Method and Device for Image Display and Eye Tracking through a Catadioptric Lens
A second HMD related granted patent published yesterday covers systems, methods, and devices for displaying an image and tracking an eye of a user through a catadioptric lens.
Apple states that when an eyepiece with a catadioptric lens is used in an HMD, and the display is moved closer to the eyepiece (and, thus, the eye), little room remains between the eyepiece and the display, making it difficult to fit an eye tracking system, such as one or more light sources and a camera. However, using a selectively distortive catadioptric lens (that distorts light from the display but does not distort light from the light sources) allows the eye tracking system to be placed very near the eyepiece without experiencing distortion from the eyepiece.
Apple's patent FIG. 11B below illustrates a block diagram of a head-mounted device having a catadioptric eyepiece with an eye tracking system disposed in front of the eyepiece in accordance with some implementations; FIG. 12 is a block diagram of an HMD device system.
(Click on image to Enlarge)
Review Apple's granted patent 11,301,036 for more technical details.
Comments