This week the European Patent Office published a patent application from Apple that relates to their XR headset being used in future science classes in schools.

Apple begins by noting that various electronic devices may provide a user with an extended reality experience that integrates rendered three-dimensional (3D) graphics (e.g., virtual objects) into a live video stream of a physical environment captured by a camera. In some instances, it may be desirable to provide computer-generated visual effects based on physical properties of objects in the physical environment to provide visualization of a knowledge domain.

For example, it may be desirable for an education experience to provide phenomenon-based thinking experiences in extended reality environments to better engage students. Phenomenon-based thinking experiences may provide different perspectives to improve understanding, provide learners with a breadth of exposure, and provide learners with an opportunity to teach each other and co-curate shared experiences in order to develop social competencies while achieving mastery and internalizing knowledge. 

Apple's patent covers devices, systems, and methods that provide an immersive experience (e.g., a learning experience) in an extended reality (XR) environment. For example, the experience may provide different presentation modes, e.g., for different knowledge/scientific domains, that provide different visualization layers to illustrate different scientific concepts.

Each visualization layer of the XR environment has virtual content that is provided based on scene understanding, e.g., which objects are wood, which are metal, etc.

For example, in an electric field domain, visualization can be provided to illustrate the effect of what Wi-Fi signals would look like if you could see them (e.g., making the invisible visible).

For example, virtual electromagnetic waves may be provided from a Wi-Fi source, and visual effects of the electromagnetic waves may be altered based on the physical properties of the objects (e.g., wood, metal, glass, etc.).

In another example, a chemistry knowledge domain may illustrate properties of objects as the objects interact in the environment. In some implementations, properties of the physical environment may be determined and used in providing the experience. For example, radiation properties of absorption, surroundings, etc., may be estimated and used to provide an experience that illustrates scientific concepts based on these properties. An immersive learning experience may be provided in an XR environment to provide a phenomena-driven learning experience that uses enhanced representation capabilities to visualize impossible learning scenarios (e.g., making the "invisible visible"), which can enable learning and improve understanding of conceptually difficult concepts.

Apple's patent Figure 3 below includes content that corresponds to features of a presentation mode for a knowledge domain associated with a particular scientific domain (e.g., optics and light refraction).

(Click on image to Enlarge)

Apple notes that the presentation mode as depicted in view #305 displays light rays #340 that are disseminated from the representation #324 (e.g., a light source, such as a central lighting fixture or light bulb in a room), and light rays #350 that are disseminated from the representation #326 (e.g., a light source, such as sunlight shining through a ceiling skylight).

For example, the presentation mode associated with view #305 may include an educational experience that illustrates for the user, in an immersive learning experience, the effect of what light rays would look like if you could see them (e.g., make the "invisible visible"). 

Virtual light rays #340 may be provided from a light source (e.g., representation #324 of light source, lightbulb #324), and visual effects of the light rays may be engaged by the user moving his or her hand (e.g., representation #328) into the light rays #340. For example, light rays #340 are absorbed by the representation #328 of the user's hand as shown at area #342. Additionally, an educational display panel #344 may be presented that graphically illustrates the wavelength of the light rays #340.

Apple's patent Figure 3 further includes another example of the scientific domain associated with optics and light refraction. For example, light rays #350 are disseminated from the representation #326 (e.g., a light source, such as sunlight shining through a ceiling skylight). The presentation mode as depicted in view 305 further illustrates an educational display panel #352 based on the light rays #350. In particular, educational display panel #352 illustrates Snell's law, a formula used to describe the relationship between the angles of incidence θ and refraction, when referring to light or other waves passing through a boundary between two different isotropic media, such as water, glass, or air (e.g., sunlight rays passing through representation #326, a skylight).

For more details, review Apple's European patent publication number 4137917. While Meta is working on the Metaverse where users will waste their time dressing up avatars and shopping, Apple is working on a series of serious uses that would appeal to both their fan base and various markets including education that will provide rich experiences to students not available today.

Apple Inventors

• Daniel Ulbricht: Senior Machine Learning R&D Manager
• Jim J. Tilander: Sr. Software Engineering Manager
• Meghan Nelson Welles: Project Lead
• Daniel Ulbricht: Senior Machine Learning R&D Manager
• Sarune Baceviciute: Research & Design Lead
• Stacey Matthias: R&D