Chipzilla in Hot Water For Cheating on Benchmarks

You may or may not know that SPEC CPU 2017 is a popular benchmark used for high-end servers and data centres, and I guess either way, it doesn’t matter if you do or not. However, the Standard Performance Evaluation Corporation or “SPEC” is designed to provide an honest and fair benchmark for these systems, and has just binned over 2600 results for the Chipzilla Xeon chips… uh-oh!

Chipzilla

The non-profit group SPEC has now said that it would stop publishing SPEC CPU 2017 results for Chipzilla PCs, which they claim have been running a modified version of the Chipzilla compiler. The group say they’ve been running a “sneaky optimisation for a specific workload” thus making their results look more favourable.

Standard Performance Evaluation Corporation

The benchmark in question is the Chipzilla API DPC++/C++ Compiler, which SPEC say has been modified for the 523.xalancbmk_r / 623.xalancbmk_s benchmarks. Jibberish to some of you I know, and largely it is to me to if I’m honest. But tinkering with software to produce higher benchmark scores is simplified enough that anyone can understand it. However, it has resulted in a 4-9% boost in the scores, which is significant.

There’s now a warning applied to thousands of results reading that “The compiler used for this result was doing a compilation that especially boosts the performance of the 523.xalancbmk_r / 623.xalancbmk_s benchmarks using insider knowledge.” Which is to say they’re completely worthless results. According to a Phoronix report, 2022.0 to 2023.0 models are affected, but similar systems from other years seem to be unaffected by the warnings.

Here is a dramatisation of the events…