Apple wins a patent for an enhanced Apple Pencil-based Controller System with added Tracking Devices for HMDs, Macs & more
Today the U.S. Patent and Trademark Office officially granted Apple a patent that relates to a possible future Apple Pencil-based hand controller system with a tracking device for HMDs, Macs, a TV (or TV box) and gaming devices. This is one of four patents supporting this project.
Tracking Devices For Handheld Controllers
Apple notes that electronic devices that are configured to be held in the hand of a user may be used to gather user input and to provide a user with output. For example, electronic devices that are configured to control one or more other electronic devices, which are sometimes referred to as controllers, handheld controllers, input devices, or handheld input devices, may be used to gather user input and to supply output or with games. If the user wishes to switch to an application that requires more precise position tracking, such as a game or creative application, the user may expand the housing of the controller.
A handheld input device may, as an example, include an inertial measurement unit with an accelerometer for gathering information on input device motions such as swiping motions, waving motions, writing movements, drawing movements, shaking motions, rotations, etc., and may include wireless communications circuitry for communicating with external equipment such as a head-mounted device, may include tracking features such as active or passive visual markers that can be tracked with an optical sensor in an external electronic device, may include input devices such as touch sensors, force sensors, buttons, knobs, wheels, etc., and may include sensors for gathering information on the interactions between the handheld input device, the user's hands interacting with the input device, and the surrounding environment. The handheld input device may include a haptic output device to provide the user's hands with haptic output and may include other output components such as one or more speakers, visual indicators, etc.
One or more handheld input devices may gather user input from a user. The user may use the handheld input devices to control a virtual reality or mixed reality device (e.g., head-mounted equipment such as glasses, goggles, a helmet, or other device with a display). During operation, the handheld input device may gather user input such as information on interactions between the handheld input device(s) and the surrounding environment, interactions between a user's fingers or hands and the surrounding environment, and interactions associated with virtual content displayed for a user.
The user input may be used in controlling visual output on a display (e.g., a head-mounted display, a computer display, etc.). Corresponding haptic output may be provided to the user's fingers using the handheld input device. Haptic output may be used, for example, to provide the fingers of a user with a desired sensation (e.g., texture, weight, torque, pushing, pulling, etc.) as the user interacts with real or virtual objects using the handheld input device.
Haptic output can also be used to create detents, to provide localized or global haptic feedback in response to user input that is supplied to the handheld input device, and/or to provide other haptic effects.
Handheld input devices can be held in one or both of a user's hands. Users can use the handheld input devices to interact with any suitable electronic equipment. For example, a user may use one or more handheld input devices to interact with a virtual reality or mixed reality system (e.g., a head-mounted device with a display), to supply input to a desktop computer, tablet computer, cellular telephone, watch, ear buds, or other accessory, to control household items such as lighting, televisions, thermostats, appliances, etc., or to interact with other electronic equipment.
A tracking device may be removably attached to a handheld input device to enhance the precision with which an external electronic device tracks the location of the handheld input device.
The tracking device may serve as a secondary input device, may have larger dimensions than the handheld input device, may have different tracking features than the handheld input device, and/or may have an expandable housing so that visual markers on the tracking device can be tracked with enhanced accuracy.
If desired, the tracking device may include input-output devices to enhance the input-output capabilities of the handheld controller or other object. The tracking device may include input-output components, haptics, sensors, a battery, and/or other circuitry and may have a recess or other engagement structure configured to attach to the tip of a handheld controller. When the tracking device is coupled to the handheld controller, the tracking device and handheld controller may form a combined handheld input device (e.g., for a head-mounted device or other electronic device), with greater tracking precision and additional input-output capabilities provided by the tracking device. When the tracking device is removed from the handheld controller, the tracking device and handheld controller may be operated independently of one another and/or may be used as two separate handheld controllers for a head-mounted device or other device.
Apple's patent FIG. 3 below is a diagram of an illustrative system including a handheld controller and an electronic device such as a head-mounted device; FIG. 4 is a diagram of an illustrative system including an electronic device such as a head-mounted device and a handheld controller coupled to a tracking device; FIG. 5 is a diagram of an illustrative system including a handheld controller and a tracking device that are decoupled from one another and are used to provide input to an external electronic device such as a head-mounted device.
Apple's patent FIG. 12 above is a perspective view of an illustrative system including a handheld controller and a tracking device that is removably coupled to the handheld controller; FIG. 15 is a perspective view of an illustrative tracking device having a telescoping housing.
To review the full details of this invention, check out granted patent 12130972. This is only one of four patents on this project with today's being the first granted patent. To get an idea of the scope of this project, check out our other reports on this: 01, 02 and 03. One set of patent figures associated with one of the patents listed above is presented below.