Today the US Patent & Trademark Office published a patent application from Apple that relates to providing users with a means of directly manipulating virtual objects presented as part of CGR environments / gaming worlds with a controlling device. This is another option that Apple is considering amongst others they've been working on such as VR Gloves (Gloves 2) or finger devices.

] Various electronic devices exist, such as head-mound devices (also known as headsets and HMDs), with displays that present users with a computer-generated reality (CGR) environment in which they may be fully immersed in a surrounding physical environment, fully immersed in a virtual reality environment comprising virtual objects, or anywhere in between.

While direct manipulation of physical objects in the surrounding physical environment is available to users naturally, the same is not true for virtual objects in the CGR environment. Lacking a means to directly interact with virtual objects presented to a user as part of a CGR environment limits a degree to which the virtual objects are integrated into the CGR environment.

Thus, it may be desirable to provide users with a means of directly manipulating virtual objects presented as part of CGR environments.

Apple's patent covers devices, systems, and methods of manipulating virtual objects using a tracked physical object while presenting content on a display of an electronic device, like a Head Mounted Device.

In one implementation, a method involves presenting content including a virtual object and a virtual representation of a proxy device physically unassociated with an electronic device on a display of the electronic device.

The proxy device includes an input device disposed on an external facing surface of a case that defines peripheral boundaries of the proxy device. The case is symmetrical with respect to each axis of a proxy device reference frame fixed at a center of the proxy device.

Input is received from the proxy device using the input device that represents a request to create a fixed alignment between the virtual object and the virtual representation in a three-dimensional ("3-D") coordinate space defined for the content.

The fixed alignment is created in response to receiving the input. A position and an orientation of the virtual object in the 3-D coordinate space is dynamically updated using position data that defines movement of the proxy device in the physical environment.

Apple's patent FIG. 1 below is a block diagram of an example operating environment; FIG. 6 illustrates an example of dynamically updating the content in response to the physical movement of the tracked physical object in the physical environment.

Apple's patent FIG. 7 above illustrates an example of a tracked physical object; and FIG. 8 illustrates another example of a tracked physical object.

Apple's patent application 20200241629 that was published today by the U.S. Patent Office was filed back in Q1 2020. Considering that this is a patent application, the timing of such a product to market is unknown at this time. The patent figure designs were common at Metaio and sure enough one of the Metaio engineers is associated with this patent.

One of the inventors listed on Apple's patent is Ryan Burgoyne, Software Development Engineer. He came to Apple via their acquisition of Metaio in May 2015.

At Metaio, Burgoyne was responsible for integrating new depth-camera tracking and reconstructing technologies developed by Metaio Research Dallas into the Metaio SDK, which was used by more than 100,000 developers worldwide.

Apple's other inventor listed is Austin Germer, Digital Artist. In his bio he notes that "I am an Environment Artist focused on creating immersive and exciting visuals in gaming. I have extensive experience in game development, driving simulation development and training.

Specialties: 3D modelling, high poly to low poly baking, texturing, optimization and integration into game engines. Other experience: Material Creation, Terrain Creation, Speed Tree Creation, Photogrammetry, Sculpting, illustration.

Germer's background is steeped in the gaming world having worked at 2K creators of Bioshock (one of my favorite games), Arkane Studios and Hangar 13.

I think it's safe to say that one aspect of the patent relates to interacting with objects in MR worlds using "wearable or handheld controllers with or without haptic feedback," as noted in the patent.