EECS 322 Computer Architecture

Benchmarks

Instructor: Francis G. Wolff wolff@eecs.cwru.edu Case Western Reserve University This presentation uses powerpoint animation: please viewshow CWRU EECS 322 1

SPEC 2000 FAQ

What is SPEC CPU2000?
Reference: http://www.specbench.org/
A non-profit group that includes computer vendors, systems integrators, universities and consultants from around the world. What do CINT2000 and CFP2000 measure? Being compute-intensive benchmarks, they measure performance of the (1) computer's processor, (2) memory architecture and (3) compiler. It is important to remember the contribution of the latter two components -- performance is more than just the processor. What is not measured? The CINT2000 and CFP2000 benchmarks do not stress: I/O (disk drives), networking or graphics.CWRU EECS 322
SPECint2000 (Number of processors = 1)
Company System Clock, CPU SPEC 298 L2 cache
256KB(I+D) 256KB(I+D) 256KB(I+D) 8M(I+D) 256KB(I+D) 256KB(I+D) 4M(I+D)
Dell Precision Ws 330 1.50 GHz P4 Dell Precision Ws 330 1.40 GHz P4 Intel VC820 SGI SGI 2200 2X Intel SE440BX-2 Intel SE440BX-2 SGI Origin200 1.13 GHz P3 400MHz R12k 800 MHz PMHz P3 360MHz R12k
Pitfall: Using MIPS or Clock speed as performance metric Pitfall: Using MIPS or Clock speed as performance metric

CWRU EECS 322 3

Doom benchmark results
Reference: http://www.complang.tuwien.ac.at/misc/doombench.html Doom,Quake games: http://www.idsoftware.com
"The Doom benchmark is more important than SPEC" "The Doom benchmark is more important than SPEC" (paraphrased) John Hennessy in his plenary talk at FCRC '99. (paraphrased) John Hennessy in his plenary talk at FCRC '99. avg. fps 304.3 201.9 197.1 196.0 190.5 188.1 Processor MIPS R4400-250 PentiumIIIE-800 PentiumIIIE-787 MIPS R10000-195 PentiumIII-644 PentiumIII-800 L1 Cache 16+16k 16+16K 16+16K 32+32k 16+16K 16+16K Mother Board SGI Indigo2 ASUS P3B-F Abit BH6R1.01 SGI Indigo2 Abit BX6 2,0 ASUS CU4VX
Wow! 250 Mhz MIPS beats the 800 Mhz Pentium. avg. fps The average number of video frames per second

CWRU EECS 322 4

Benchmark wars: Internet Servers
Sm@rt Reseller's January 1999 article, Linux Is The Web Servers Choicesaid "Linux with Apache beats NT 4.0 with IIS, hands down."
http://www.kegel.com/nt-linux-benchmarks.html
In March 1999, Microsoft commissioned Mindcraft to carry out a comparison between NT and Linux.

CWRU EECS 322 5

Benchmark Wars: Linux/Solaris
PC Magazine, September 1999 Sun Microsystems SPARC architecture now jumps in!
.found that NT did a lot more disk accesses than Linux, which let Linux score about 50% better than NT.

CWRU EECS 322 6

Performance
To maximize performance, we want to minimize response time or execution time Performance = 1 Execution time
To compare the relative performance, n, between machine X and Y, we use PerformanceX = PerformanceY Execution timeX

CWRU EECS 322 7

Execution timeY =n

Measuring Performance

Total program clock cycles executed Execution time = Clock frequency rate (MHz) Total program instructions exec x CPI = Clock frequency rate (MHz)

CPI = Average number of clock cycles per instruction

Clock cycle time (us) =

1 Clock frequency rate (Mhz)

CWRU EECS 322 8

CPI Example
Given the following instruction class execution times: alu=6ns, loads=8ns, stores=7ns, branches=5ns, jumps=2ns CPI = (6ns+8ns+7ns+5ns+2ns)/5 = 28/5 = 5.6 ns = (0.2*6ns+0.2*8ns+0.2*7ns+0.2*5ns+0.2*2ns) = 5.6 ns
Given the following instruction class execution times: alu=60%, loads=20%, stores=10%, branches=5%, jumps=5% alu=6ns, loads=8ns, stores=7ns, branches=5ns, jumps=2ns CPI = (0.6*6ns+0.2*8ns+0.1*7ns+0.05*5ns+0.05*2ns) = 6.25

CWRU EECS 322 9

Performance example
Benchmark A Benchmark A B B L Total L Total 2 ====6

(PH page 64)

Instruction class Instruction class ALU ALU Branches Branches Load/Stores Load/Stores

CPI CPI 3 3

Total CPU cycles1 = (2xA) + (1xB) + (2xL) = (2x1) + (1x2) + (2x3) = 10 cycles CPI1 = 10 cycles/5 = 2 average cycles per instruction Total CPU cycles2 = (4x1) + (1x2) + (1x3) = 9 cycles CPI2 = 9 cycles/6 = 1.5 average cycles per instruction Benchmark 2 executed more instructions, but was faster.

CWRU EECS 322 10

MIPS Performance example
Benchmark A B Benchmark A B Compiler 1 5xCompiler 1 5xCompiler 1099 Compiler 10 L L Total Total =7x1099 =7x10 =12x1099 =12x10

(PH page 78)

Total CPU cycles1 = (5xA) + (1xB) + (1xL) = 10x109 cycles Execution time1 = 10x109 cycles/500Mhz = 20 seconds CPI1 = 10x109 cycles/ 7x109 = 1.43 MIPS1 = Clock rate/CPI = 500Mhz/1.43 = 350 MIPS Total CPU cycles2 = (10xA)+(1xB)+(1xL) = 15x109 cycles Execution time2 = 15x109 cycles/500Mhz = 30 seconds CPI2 = 15x109 cycles/12x109=1.25 MIPS2= 500Mhz/1.25 = 400 MIPS Although MIPS2 > MIPS1 but execution time is unexpected!322 CWRU EECS
Amdahls Law (the law of dimishing returns)
Execution Time After Improvement = Execution Time Unaffected + (Execution Time Affected / Amount of Improvement) Example: "Suppose a program runs in 100 seconds on a machine, with multiply responsible for 80 seconds of this time. How much do we have to improve the speed of multiplication if we want the program to run 4 times faster?" How about making it 5 times faster? Principle: Make the common case fast Well, lets speed up the multiply!CWRU EECS 322

Execution Time After Improvement = (Execution Time Affected / Amount of Improvement) + Execution Time Unaffected Let Execution Time After Improvement be old time / speed up = 100 seconds / 5 times faster = 20 seconds = Execution Time needed = 80 seconds/n + (100-80 seconds) Equating both sides 20 = 80 seconds/n 0 = 80 seconds/n + (100-80 seconds)
No amount of multiplier speed up can make a 5 fold increase

CWRU EECS 322 13

Sources of improvement
For a given instruction set architecture, increases in CPU performance can come from three sources 1. Increase the clock rate 2. Improve the hardware organization that lower the CPI 3. Compiler enhancements that lower the instruction count or generate instructions with a lower average CPI In addition to the above, in order to improve CPU efficiency of software benchmarks. Improve the software organization (data structures, )

CWRU EECS 322 14

Performance Summary
Execution time is the only valid and unimpeachable measure of performance. Any measure that summarizes performance should reflect execution time. Designers must balance high-performance with low-cost. You should not always believe everything you read! Read carefully! (see newspaper articles, e.g., Exercise 2.37)

CWRU EECS 322 15