On Monday a consortium involving more than 40 leading companies and university researchers introduced MLPerf Inference v0.5, the first industry standard machine learning benchmark suite for measuring system performance and power efficiency.

The mission of MLPerf consortium is to deliver, as with v0.5, fair and useful benchmarks for measuring training and inference performance of Machine Learning (ML) hardware, software, and services. We believe that a widely accepted benchmark suite will benefit the entire community, including researchers, developers, hardware manufacturers, builders of machine learning frameworks, cloud service providers, application providers, and end users.

In the consortium's press release they note that the benchmark suite covers models applicable to a wide range of applications including autonomous driving and natural language processing, on a variety of form factors, including smartphones, PCs, edge servers, and cloud computing platforms in the data center. MLPerf Inference v0.5 uses a combination of carefully selected models and data sets to ensure that the results are relevant to real-world applications. It will stimulate innovation within the academic and research communities and push the state-of-the-art forward.

By measuring inference, this benchmark suite will give valuable information on how quickly a trained neural network can process new data to provide useful insights. Previously, MLPerf released the companion Training v0.5 benchmark suite leading to 29 different results measuring the performance of cutting-edge systems for training deep neural networks.

MLPerf Inference v0.5 consists of five benchmarks, focused on three common ML tasks:

• Image Classification - predicting a “label” for a given image from the ImageNet dataset, such as identifying items in a photo.

• Object Detection - picking out an object using a bounding box within an image from the MS-COCO dataset, commonly used in robotics, automation, and automotive.

• Machine Translation - translating sentences between English and German using the WMT English-German benchmark, similar to auto-translate features in widely used chat and email applications.

The current list of companies and researchers participating in the consortium is presented in the graphic below. Although Apple is not listed in this initial list, it doesn't mean that they won't be a participant in the future. The consortium notes that their list will be updated in the not-too-distant future.

The Scribd document below provides you with a grand overview of the MLPerf consortium's vision and is provided courtesy of Patently Apple.

The Vision behind MLPerf

   MLPerf - Vision Behind MLPerf by Jack Purcher on Scribd

Apple could be a major contributor to the MLPerf consortium. At present, Apple continues to publish Machine Learning articles within its ML Journal; develops ML tools for their developers; promotes Machine Learning jobs at Apple and presents a series of videos about the ML work at Apple and on June 20th listed a job for an Applied Research Scientist - Machine Learning for their new Seattle facility. Apple is looking for someone with experience in the health domain. Apple also integrated Machine Learning into their A12 Bionic processor. Apple-designed Neural Engine is built for advanced, real-time machine learning.

About Making Comments on our Site: Patently Apple reserves the right to post, dismiss or edit any comments. Those using abusive language or negative behavior will result in being blacklisted on Disqus.