Apple's 3D Object Recognition Patent

Apple's invention consists of a statistical model of the shape variations in a class of objects relating the two-dimensional (2D) projection in images to the three-dimensional (3D) shape of the object and the use of the 3D shape information for identification or verification of the object. Furthermore, the present invention relates to an image processing device or system for implementing such a method. The process is fully automatic and may be used e.g. for biometric identification from face images or identification of objects in for instance airport security X-ray images. The recovered 3D shape is the most probable shape consistent with the 2D projections, i.e. the images. The statistical model needs a bank of data, denoted training data, where the 3D positions of the image features are known, in order to learn the parameters of the model. Such data sampling can be done using e.g. binocular or multi-view stereo or range scanners. Once the model parameters are learned, the 3D shape can be computed using one or several images. The 3D shape is then used, by means of the presented invention together with the 2D image data, to identify or verify the object as a particular instance of the object class, e.g. the face belonging to a certain individual. A positive (or negative) identification initiate proper action by means of the presented innovation.

Patent FIGS. 1A through to 1C illustrate a two-step procedure for recovering 3D data from an input image; FIGS. 2B and 2C illustrate a process of surface fitting to a recovered 3D shape.

Medical Market

Medical markets use advanced 3D modeling and Apple may be focusing on advancing it further in the future. In this particular area, Apple's patents states that "once the 3D data is recovered, a surface model could be fitted to the 3D structure. This might be desirable in case the two-step procedure above only produces a sparse set of features in 3D space such as e.g. points-and space curves. Even if these cues are characteristic for a particular sample (or individual), it is often not enough to infer a complete surface model, and in particular, this is difficult in the regions where the features are sparse. Therefore, a 3D surface model consisting of the complete mean surface is introduced. This will serve as a domain-specific, i.e. specific for a certain class of objects, regularizer. This approach requires that there is dense 3D shape information available for some training examples in the training data of the object class obtained from e.g. laser scans or in the case of medical images from e.g. MRI or computer tomography. From these dense 3D shapes, a model can be built separate from the feature model above. This means that, given recovered 3D shape, in the form of points and curves, from the feature model, the best dense shape according to the recovered 3D shape can be computed. This dense shape information can be used to improve surface fitting.

As a second example, consider the case of the object class being a particular blood vessel, e.g. the aorta. The model is then learned using e.g. curves and contours in images together with the true 3D shape obtained as e.g. a 3D MRI image. From the true 3D shapes a second model is learned comprising of the surface of the aorta. Then the most probable (or highly probable) aorta surface can be recovered from the image features or from the 3D shape recovered by the primary shape model.

Below is a video of hospital in Calgary, Alberta, Canada that uses the iPhone for 3D medical imagery.

It should be noted that Apple has already been granted a patent regarding this technology. In today's patent application, Apple cancels 24 previous patent claims and presents eighteen new claims to protect this technology. This is a good sign that Apple wants to use this technology and wishes to further protect it. The history behind this patent is very interesting and is outlined in our granted patent report noted above.

Big Day for FileMaker

This is by far the biggest single patent day for FileMaker that I could remember, and with FileMaker's Break-Through Conference fast approaching, there may be some new products and upgrades relating to today's patent applications.

FileMaker: Patents Relating to Database Architecture

One: Extended Database Search

Apple's Patent Abstract: Methods, program products, and systems implementing integrated repository of structured and unstructured data are disclosed. An integrated environment is provided for accessing, structured data (e.g., data of a relational database) and unstructured data (e.g., data stored in a text or binary file), including creating, managing, modifying, and searching the structured data and unstructured data. The integrated environment can include an integrated user interface, a set of commands and application programming interface (API), and storage for a relational database and a document repository. The integrated environment can include a database abstraction layer that allows database operations on both the structured data and the unstructured data. For More information, see patent application 20120117116.

Two: Integrated Repository of Structured and Unstructured Data

Apple's Patent Abstract: Methods, program products, and systems implementing integrated repository of structured and unstructured data are disclosed. An integrated environment is provided for accessing, structured data (e.g., data of a relational database) and unstructured data (e.g., data stored in a text or binary file), including creating, managing, modifying, and searching the structured data and unstructured data. The integrated environment can include an integrated user interface, a set of commands and application programming interface (API), and storage for a relational database and a document repository. The integrated environment can include a database abstraction layer that allows database operations on both the structured data and the unstructured data. For more information, see patent application 20120117120

Three: Browser-Based Database Manipulation

Apple's Patent Abstract: Methods, program products, and systems implementing browser-based database manipulation are described. A browser on a client device can receive a form definition that is generated based on a schema of a relational database and a layout. The schema in the form definition can specify properties of data fields. The browser can display the data fields and other components of the layout in a list. The browser can receive a drag-and-drop input, the drag-and-drop input selecting a data field or a layout component from the list and dropping the selected data field or layout component in a drop zone in a form area in the browser. The browser can determine the drop location, and use the drop location as a parameter to modify the form definition. The layout and the schema can be modified based on the modified form definition. For more information, see patent application 20120117121.

Four: Dynamic Data Association

Apple's Patent Abstract: Methods, program products, and systems implementing dynamic data association are described. Relations between a first data item and a second data item can be created in an ad hoc manner. The first data item can include a row of data in a table in a relational database. The second data item can include a row of data in the same table, in another table in the relational database, or a document outside the relational database. Configurable metadata can be created to link the first data item and the second data item. The link can be directional. The configurable metadata can include data specified by a user. The configurable metadata can be stored in one or more system tables in the relational database. The configurable metadata can be specific to the user, or to a work group of the user. The configurable metadata can change when the first data item or the second data item is modified. For more information, see patent application 20120117462.

FileMaker: Patents Relating to Data Management

One: Intelligent Data Caching

Apple's Patent Abstract: Methods, program products, and systems implementing intelligent data caching are disclosed. A client can send a request for data to a server. The request can include a query and a query context. The query can retrieve more data than can be displayed in a current page. The query context can specify a quantity of data records that can be displayed in the current page. The client can receive from the server two sets of data in response to the request. The first set of data can include row identifiers of data records that satisfy the query. The second set of data can include data records that can be displayed in the current page. The client can store the two sets of data in one or more buffers (e.g., two buffers) and update buffer(s) when different data records are displayed (e.g., when a user scrolls through pages). For more information, see patent application 20120117509

Two: Collaborative Database Operations

Apple's Patent Abstract: Methods, program products, and systems implementing collaborative database operations are disclosed. Collaborative database operations can be facilitated by data notification, where a server notifies a client when data being viewed at the client are updated. Notification can be achieved through registration and polling. A client of a database can request data from a server. The request can include a data request and a registration request. The data request can include a query into a database. The registration request can include a specification on conditions upon which the client is to be notified. The condition can include an event type. Upon registration, the client can poll the server periodically to determine whether a result of the query has changed, and if so, what data record in the result has changed. If a data record has changed, the client can request the changed data using an identifier of the data record. For more information, see patent application 20120117105.

Notice

Patently Apple presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. Apple's patent applications have provided the Mac community with a clear heads-up on some of Apple's greatest product trends including the iPod, iPhone, iPad, iOS cameras, LED displays, iCloud services for iTunes and more. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.

Check out Our Latest Reports on Patent Bolt Titled:

Microsoft Illustrates Reinvented Xbox TV Remote & Controller

Microsoft Invents Portable Devices with Transparent Displays