The Transaction Processing Performance Council (TPC) has launched the first industry-standard, vendor-neutral benchmark for measuring real-world, end-to-end AI and machine learning (ML) scenarios.
The TPCx-AI benchmarks uses a diverse dataset and was specifically designed to be adaptable across a wide range of different systems from the edge to the data centre. This should allow a comparison of edge AI chips, and data centre chips and FPGA acclerators for real world applications.
“The TPCx-AI benchmark is the result of collaboration between talented engineers and researchers at some of today’s leading AI organizations,” said Hamesh Patel, chair of the TPCx-AI committee and principal engineer at Intel Corporation. “It is designed to emulate real-world examples of organizations that use a variety of production ready data science pipelines – including both AI and ML approaches – and is now widely available to anyone who would like to download and run it. We look forward to feedback as industry experts, academics and others interested in benchmarking system performance begin to use it.”
The TPCx-AI benchmark provides a means to evaluate performance for the System Under Test (SUT) as a general-purpose data science system that generates and processes large volumes of data and trains preprocessed data to produce realistic machine learning models. The benchmark allows for flexibility in configuration changes.
The benchmark was developed by a group that includes Intel, IBM, Cisco, Dell, HPE, Microsoft, Red Hat, TTA, and VMware.
The benchmark also measures the end-to-end time to provide insights for individual use cases, as well as throughput metrics to simulate multiuser environments for a given hardware, operating system, and data processing system configuration under a controlled, complex, multi-user AI or machine learning data science workload.
Other members of the TPC include Actian, Alibaba, AMD, Fujitsu, Hitachi, Huawei, Lenovo, Nvidia and Oracle.
As an “Express” benchmark, TPCx-AI is an executable kit that can be rapidly deployed and measured. It is designed to provide relevant, objective performance data to industry users and is available for download at
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.