Patent wise, Apple's #1 project in 2020 has clearly centered on their future Headset, with over 25 patents on this since January 2020 alone. On Thursday the US Patent & Trademark Office published a patent application from Apple that relates to 3D modeling of fingers to be used in Mixed Reality Headsets for object and gesture recognition.

Today's patent report also covers a peek at some of the patents that NextVR holds should Apple acquire this VR focused company as a recent rumor suggested.

Generating a 3D Model of a Fingertip for Visual Touch Detection

Apple's patent in-part describes generating a 3D model of fingers that could be used for touching virtual objects and for in-air gesturing for the headset to understand commands.

Apple notes in patent point #31 that "a user may spin a finger in front of a camera in order interact with a physical environment or to sign into an account" and so forth.

Apple's patent FIG. 4 presents a flow chart of an example technique for utilizing a 3D model of a fingertip for gesture recognition; FIG. 5 shows an example system diagram illustrating a system setup for generating a 3D model of a fingertip.

Further to FIG. 4, The hand model may include 3D models of a hand corresponding to various poses of the hand. The flowchart continues at #410, and the electronic device 100 (iPhone) updates the 3D model of the hand based on the first and second image. For example, a depth image may be obtained from a perspective such that a touch point between the touching object and the surface is occluded by the touching object (e.g., the fingertip touching a surface may be occluded by the hand from the point of view of the user).

The model of the touching object may be generated or refined based on the combined data from the touch images and/or depth data. The additional information regarding the finger may be utilized to update or refine a 3D model of a hand that includes the finger.

In one or more embodiments, ICP (iterative closet point) algorithms, contour alignment-based algorithms or any algorithm based on mutual information could be used to fit the hand model or finger model with an image of the hand or finger to improve or enable hand tracking or touch detection.

Apple's patent covers both recognizing finger gestures and visual touch detection.

Apple's patent application 20200105055 that was published last Thursday by the U.S. Patent Office was filed back in Q3 2019 that includes IP from a 2018 patent never made public. Considering that this is a patent application, the timing of such a product to market is unknown at this time.

Apple Inventors

Rohit Sethi: Engineering Manager; Deep Learning and Computer Vision Algorithms. Previously worked at NVIDIA.

Lejing Wang: Tech Lead / Engineering Manager; Came to Apple via Metaio acquisition 5 years ago.

Headset Rumor

In other headset news, Apple is reportedly looking to acquire virtual reality company NextVR. According to 9to5Mac's rumor report: "The icing on the cake may not be expertise in virtual reality, however, as NextVR also has holds patented technology that upscales video streams. NextVR uses this technology to support high quality video streams of music and sporting events to VR headsets. NextVR holds over 40 technology patents in total."

Some of their VR technology is a little game-like as you'll see in the NBA game video below where at times the ball appears like a burning comet or ball of fire or a lightning bolt.

Beyond streaming content, NextVR has some agreements in hand with the NBA and NFL which would be of great interest to Apple making such VR content a major draw for their future headset. 

One NextVR's granted patent 10,298,819 covers a 3D camera. NextVR patent FIG. 20 below is a drawing of an exemplary stereoscopic camera (3D camera); FIG. 31 is a drawing illustrating the exemplary camera apparatus with the sensor assembly cover plate.

Below is a sampling of ten other NextVR patents on record. In total Patently Apple was only able to discover 38 patents on record and not 40+. They have many headset related patents and even one relating to a VR headset playing in vehicles which may interest Apple regarding their Project Titan project. We covered one Apple headset patent relating this in Dec. 2019.

Methods and apparatus for streaming content

Patent number: 9699437

Abstract: Methods and apparatus for streaming content corresponding to a 360-degree field of view are described. The methods and apparatus of the present invention are well suited for use with 3D immersion systems and/or head mounted display which allow a user to turn his or her head and see a corresponding scene portion. The methods and apparatus can support real or near real time streaming of 3D image content corresponding to a 360-degree field of view.

Methods and apparatus for delivering content and/or playing back content

Patent number: 9918136

Abstract: Content delivery and playback methods and apparatus are described. The methods and apparatus are well suited for delivery and playback of content corresponding to a 360-degree environment and can be used to support streaming and/or real time delivery of 3D content corresponding to an event, e.g., while the event is ongoing or after the event is over. Portions of the environment are captured by cameras located at different positions. The content captured from different locations is encoded and made available for delivery. A playback device selects the content to be received based in a user's head position.

Methods and apparatus for compensating for vehicular motion

Patent number: 9821920

Abstract: A head mounted virtual reality (VR) device including an inertial measurement unit (IMU) is located in a vehicle which may be, and sometimes is, moving. Detected motion attributable to vehicle motion is filtered out based on one or more or all of: vehicle type information, information derived from sensors located in the vehicle external to the head mounted VR device, and/or captured images including a reference point or reference object within the vehicle. An image portion of a simulated VR environment is selected and presented to the user of the head mounted VR device based on the filtered motion information. Thus, the image portion presented to the user of the head mounted VR device is substantially unaffected by vehicle motion and corresponds to user induced head motion.

Immersive stereoscopic video acquisition, encoding and virtual reality playback methods and apparatus

Patent number: 9538160

Abstract: Methods and apparatus for capturing and displaying stereoscopic images are described in a manner that allows a user to obtain a 3d virtual reality experience simulating that of being in a seat at a football game or other event. Rear images are modified, e.g., in luminance intensity, to make them consistent with the luminance intensity of the forward images to avoid or reduce edges or differences in luminance intensity as a users turns his head from viewing a main image area to a side or rear image area. A seamless 3D presentation is made possible through the use of fisheye lenses at capture time and combining of images corresponding to forward and rear image areas as a user turns his or her head requiring a change in the captured image area which is displayed to the user.

Calibration for immersive content systems

Patent number: 9865055

Abstract: Camera and/or lens calibration information is generated as part of a calibration process in video systems including 3-dimensional (3D) immersive content systems. The calibration information can be used to correct for distortions associated with the source camera and/or lens. A calibration profile can include information sufficient to allow the system to correct for camera and/or lens distortion/variation. This can be accomplished by capturing a calibration image of a physical 3D object corresponding to the simulated 3D environment, and creating the calibration profile by processing the calibration image. The calibration profile can then be used to project the source content directly into the 3D viewing space while also accounting for distortion/variation, and without first translating into an intermediate space (e.g., a rectilinear space) to account for lens distortion.

Methods and apparatus for capturing, streaming and/or playing back content

Patent number: 10397543

Abstract: Methods and apparatus for streaming or playing back stereoscopic content are described. Camera dependent correction information is communicated to a playback device and applied in the playback device to compensate for distortions introduced by the lenses of individual cameras. By performing lens dependent distortion compensation in the playback device edges which might be lost if correction were performed prior to encoding are preserved. Distortion correction information maybe in the form of UV map correction information. The correction information may indicate changes to be made to information in a UV map, e.g., at rendering time, to compensate for distortions specific to an individual camera. Different sets of correction information maybe communicated and used for different cameras of a stereoscopic pair which provide images that are rendered using the same UV map. The communicated correction information is sometimes called a correction mesh since it is used to correct mesh related information.

Patent number: 9832449: Methods and apparatus for controlling a viewing position

Patent number: 9836845: Methods and apparatus for detecting objects in proximity to a viewer and presenting visual representations of objects in a simulated environment

Patent number: 10523920: Stereoscopic image processing methods and apparatus

Patent number: 10477186: Methods and apparatus for calibrating and/or adjusting the arrangement of cameras in a camera pair.