The U.S. Patent and Trademark Office officially published a series of 43 newly granted patents for Apple Inc. today. In this particular report we cover two iDevice camera inventions. The first covers 3D photography using advanced LIDAR technology for iDevices, the same technology that's used in taking street map shots. It's a scanning method that has many applications including advanced layered photo editing and creating an avatar using Apple's recently acquired Faceshift. This technology is likely to be applied when Apple introduces their future dual lens cameras for both iOS and OS X. The second camera patent relates to users having the ability to turn on a Super Resolution Mode on their iOS devices to support taking higher quality Panorama shots.

Granted Patent: 3D Depth Point Cloud from Timing Flight of 2D Scanned Light Beam Pulses

Apple's newly granted patent generally relates to an optical remote sensing system that can automatically produce a digital, time-varying 3-dimensional (3D) point cloud that represents the 3D coordinates of real world surfaces that are within a given field of view (FOV) and a given distance or depth. In one embodiment, the system can be described as a scanning LIDAR system. The system has an emitter that produces pulses of coherent collimated light beams (e.g., laser pulses), a mirror system, and a detector that is substantially co-located with the emitter to detect the pulsed light beams when they have been scattered or reflected by an object in the scene.

While LIDAR systems are used to create street maps, like the kind Apple is now in the process of gathering (also see Apple Maps Vehicles list), Apple's patent states that the new LIDAR system could be "within the same housing of a larger host device such as a consumer electronics product." For context as to what Apple could eventually use this invention for, see our 2012 report titled "Apple Invents a Killer 3D Imaging Camera for iOS Devices," which covered advanced 3D imaging using LIDAR and beyond. In that report we learned that Apple's invention related to "systems, apparatuses and methods for capturing a three-dimensional image using one or more dedicated cameras." This is why we noted in our opening summary that this invention could apply to future iOS devices using dual cameras.

Apple's patent FIG. 1 is a block diagram of a scanning LIDAR system; FIG. 2 illustrates how reflected light pulses are captured by a photodetector in the LIDAR system, and illustrates the x-axis FOV.

What Apps will use this Technology?

What will this new 3D scanning technology achieve? Some of the applications for this technology could include better interaction with an app like Faceshift that Apple has reportedly acquired. A 2015 Tom's Guide report covers the apps that could apply to Intel's RealSense 3D camera. Apple's dual lens 3D camera solution will offer similar capabilities. Such a system could also apply to advanced in-air gesturing systems. Apple was just granted such a patent earlier today and has numerous others on record.

Apple's system could also apply to 'photo layer editing and refocusing; 3D gaming with Oculus-like devices; auto login using advanced face recognition; assistance with self-driving vehicles; and object avoidance for consumer drones. 

Intel also points to 3D immersion on this RealSense webpage that states: "Scan real-life things - like a piece of art, a child's toy, or your own face and create a digital 3D version that you can share to your favorite social media site or print to a 3D printer" ... or use as an avatar in a game as Hasbro described in their recent patent filing.

Apple's granted patent 9,285,477 was originally filed in Q1 2014 and published today by the US Patent and Trademark Office.

Granted Patent: Super-Resolution based on Optical Image Stabilization

Apple's newly granted patent generally relates to camera system within consumer electronic devices like the iPhone and iPad and, particularly, to one having an optical image stabilization mechanism to vary an optical path to a sensor of the camera.

Patently Apple first covered this patent in an in-depth patent application report titled "Apple Invents New iDevice Super-Resolution Camera Engine." While the patent doesn't mention it specifically, Apple's patent FIG. 8 below clearly shows that this could be associated with taking panoramic shots.

Apple notes that "to counteract distortion from motion while capturing an image (e.g., jittering by a user operating a camera), a camera may include an optical image stabilization (OIS) system. Such an OIS system is operable to compensate for camera motion during image capture to minimize distortion of the captured image. The OIS system accomplishes this compensation by varying the optical path to the sensor in response to movement of the camera, such as jittering by the user operating the camera. This OIS system may be implemented by translating the lens itself, or may be implemented by moving an entire module that includes both lens and sensor to compensate for the movement of the camera. In any implementation, it is paramount that the OIS system stabilizes the image projected on the sensor before the image captured by the sensor is converted into digital information."

Technically speaking, Apple's patent claim 24 covers the super-resolution aspect of the invention as follows:

"A method for capturing a super-resolution image by an image capturing device, comprising: capturing, using an image sensor, a plurality of optical samples along a plurality of optical paths obtained by signaling an actuator to move a lens located in front of the image sensor so as to stabilize an optical image that is being projected onto the sensor through an optical path, before the sensor converts the optical image into digital form being said captured plurality of optical samples, in response to using an accelerometer to detect movement of the device due to involuntary shaking by a user who is holding the device; commanding a plurality of calibrated super resolution shifts of the optical path to the image sensor by an optical image stabilization (OIS) processor, wherein the plurality of calibrated super resolution shifts are associated with sub-pixel offsets, and wherein the command of the plurality of calibrated super resolution shifts comprises signaling the actuator to move the lens while the plurality of optical samples are being captured; combining the plurality of optical samples to create a densely sampled image; commanding a plurality of calibrated increase color sampling density shifts of the optical path to the image sensor by an optical image stabilization (OIS) processor, wherein the plurality of calibrated increase color sampling density shifts are associated with one-pixel offsets to provide full color sampling at each pixel location of each color the image sensor is capable of capturing, and wherein the command of the plurality of calibrated increase color sampling density shifts comprises signaling the actuator to move the lens while the plurality of optical samples are being captured; and combining the plurality of optical samples to create a increase color sampling densely sampled image.

The principle of combining multiple photos into one isn't new, but shifting the image on the sensor between shots is.

Apple filed granted patent 9,288,395 back in Q4 2012.

The Remaining Patents granted to Apple Today

Note: In order to see a clearer image of the list above, simply click on the image above to enlarge it. Some browsers may require that you click on the image and then a second click on the image to enlarge it fully.

Patently Apple presents only a brief summary of granted patents with associated graphics for journalistic news purposes as each Granted Patent is revealed by the U.S. Patent & Trademark Office. Readers are cautioned that the full text of any Granted Patent should be read in its entirety for full details. About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Comments are reviewed daily from 5am to 6pm MST and sporadically over the weekend.