The SPEC benchmarks present results with respect to a standard machine by normalizing the benchmarks. That is, a set of benchmarks is run on a reference machine and the times obtained for each of the benchmarks. When a test machine is benchmarked, its times are divided by the results on the reference machine. What are the advantages and disadvantages of giving benchmarks with respect to a reference machine?

Question

What will be an ideal response?

Answer 1

Suppose benchmarks did not normalize their results with respect to a standard machine. The time to run three programs might be 3s, 20s and 100s. How do we average these results? As they are times, we could take the average time and say it is (3 + 20 + 100)/3 = 41s. The problem is that this average depends on the size of the three programs. If one program is particularly big, it will dominate the average and benchmarks will become a reflection of how fast this large program runs on the various machines. In other words, one program is dominating the test.


Normalizing solves the problem of the domination of one program, because now all programs are related to the time run on a standard machine. Suppose that a standard machine takes 2, 5, 100 seconds to run three benchmarks and that the test machine takes 1, 2.5, 40 seconds. If we normalize times with respect to the standard we get 1/2, 2.5/5, 40/100 = 0.5, 0.5, 0.4. Now you can see that the test machine is twice as fast at executing two tasks and 2.5 times as fast on one task.