Today the U.S. Patent and Trademark Office officially published a patent application from Apple that relates to electronic devices having systems for outputting computer generated imagery. It would appear to be an Generative AI related patent, especially in light of the lead engineer on the project being Oncel Tuzel: Distinguished Scientist, Machine Leaning Research. He notes on his LinkedIn profile: "Yesterday we released OpenELM (Open LLM). Today we released CatLIP (Open vision FM)" – terms related to Generative AI.

Apple notes in their patent background that some electronic devices can include systems for generating photorealistic images of a 3-dimensional object such as images of a face. Three-dimensional (3D) generative frameworks have been developed that leverage state-of-the-art two-dimensional convolution neural network based image generators to generate realistic images of a human face. Existing 3D generative frameworks model the geometry, appearance, and color of an object or scene captured from an initial viewpoint and are capable of rendering a new image of the object or scene from a new viewpoint. However, the existing 3D generative frameworks are not able to render new images under different lighting conditions.

Image Generation System With Controllable Scene Lighting

An electronic device can be provided with a light based image generation system capable to generating photorealistic images of three-dimensional (3D) objects such as images of a face. The light based image generation system can generate images of any given 3D object under different scene/ambient lighting conditions and from different perspectives.

A method of generating an image of an object in a scene may include receiving lighting information about the scene and a perspective of the object in the scene, extracting corresponding features based on the received lighting information and perspective, decoding diffuse and specular reflection parameters based on the extracted features, and rendering a set of images based on the decoded diffuse and specular reflection parameters. The extracted features can be triplane features. The set of images may be obtained using volume rendering operations.

A method of operating an image generation system to generate an image of a 3D object may include capturing an image of the 3D object using a camera, conditioning the image generation system to generate an image of the 3D object, and generating images of the 3D object under different lighting conditions based on a trained model that uses diffuse and specular lighting parameters. The diffuse and specular lighting parameters may be decoded separately. The trained model may be a 3D generative model that is trained using unlabeled ground truth images of real human faces. The image generated system can be used to generate an avatar of a user under different environment lighting conditions and under different poses.

A method of training a light based image generation system may include receiving lighting and pose information and extracting corresponding features, obtaining diffuse and specular parameters based on the extracted features, rendering a set of images based on the diffuse and specular parameters, generating a super resolution image from the set of images, and comparing the super resolution image to one or more ground truth images to adjust weights associates with the light based image generation system.

Apple's patent FIG. 1 below is a schematic diagram of an illustrative electronic device with a "Light-based Image Generator" and "Position and motion sensors.":  FIG. 9 is a flow chart of illustrative steps for using the light based image generation system.

Apple's patent FIG. 3 above is a diagram showing a front view of an illustrative three-dimensional (3D) object under a first scene lighting condition; FIG. 4 is a diagram showing a front view of an illustrative three-dimensional (3D) object under a second scene lighting condition; FIG. 5 is a diagram showing a front view of an illustrative three-dimensional (3D) object under a third scene lighting condition; FIG. 9 is a flow chart of illustrative steps for using the light based image generation system.

To review the full details of this invention, check out patent application 20240153201. Whether this relates to a stand alone application or part of a larger Generative AI visual app is unknown at this time.