Project Goals
Establish a resource for researchers on campus with large computing needs. Help researchers convert their programs to run on the cluster. Research performance bottlenecks. Develop tools to improve the usability of clusters.
The Beowulf Cluster Lab is funded by the National Science Foundation Major Research Infrastructure Award No. 0321233.

Cluster Specifications

The main Beowulf Cluster (beowulf.boisestate.edu)
61 nodes 122 2.4 GHz Intel Xeon processors 64GB RAM 2.4 TB disk space private Gigabit network Gigabit connection to the campus backbone
6 processor developmental cluster (tux.boisestate.edu) 32 processor teaching cluster (onyx.boisestate.edu)

Other clusters:

Beowulf Cluster Lab

Cluster Hardware

Compute nodes (about $1400/node for 64 nodes = $90,000)
Tyan i7505 S2665ANF dual-533 MHz FSB dual 2.4 Ghz Intel Xeon CPUs with 512K Cache 533MHz FSB 2 x 512MB Micron Technology Memory Module 184-pin DIMM PC2100 DDR 266 MHz, unbuffered, non-parity Samsung SP4002H disk drives, 80GB 7200RPM ATA100 HP Broadcom NetXtreme 5782 Gigabit card Antec 1080 Plus AMG case with Antec True Power 550W Supply Master node: same, except with 4GB RAM and SATA drives with RAID
Networking: (about $12,000)
3 x Cisco 3750G 24-port Stacking Cluster Gigabit Switches with redundant power supply
Facilities: Liebert A/C, power setup to handle up to 300 Amps ($29,000)

Cluster Software

Red Hat Linux 9.0 with custom 2.4.24 -bigmem SMP kernel (Fedora Core 1 Linux with stock kernel on the cluster used for teaching) Portable Batch Scheduling for job scheduling Parallel Programming Libraries and Tools PVM, MPICH/MPI, LAM/MPI XPVM and XMPI Portland Group Cluster Development Toolkit HPF, Fortran 90, Fortran 77, C, C++ Parallel graphical debugger Parallel graphical profiler GNU C, C++ and Fortran 77 compilers and related tool set like ddd (Data Display Debugger) Full suite of other tools available under Linux.
Cluster Setup Experiences
YACI (Yet Another Cluster Installer) was used for automated installation. The 61- node cluster went from bare disks to fully operational in 12 minutes! YACI is available from Larwence Livermore National Lab. Design choice to go with boxes instead of blades since cooling boxes is easier and real estate was a relatively smaller issue. Evaluated AMD Athlon, AMD Opteron, Intel Xeon for Performance/Power/Price (PPP) factor to choose Intel Xeons. Chose to go with a regular PC assembler rather than a cluster company to keep costs down and have more control of what goes in each node.

People

Faculty: Amit Jain (Computer Science) and Paul Michaels (Geophysics) Graduate Students: Kevin Nuss, Hongyi Hu and Mason Vail Undergraduate Students: Joey Mazzarelli, Brady Catherman,
Luke Hindman, Charles Paulson, Jason Main and Oralee Nudson.
The project uses a model of teaming up computer scientists with researchers from other fields to create a synergistic environment.

Projects

Some applications running on the cluster.
Air Quality Modeling. Paul Dawson (Mechanical Engineering), Kevin
Nuss and Charles Paulson.
Modeling of Ocean Currents. Jodi Mead (Mathematics) and Hongyi Hu. Waveform Relaxation. Barbara Zubik-Kowal (Mathematics) and Hongyi
Hydraulic Tomography. Tom Clemo (Geophysics) and Kevin Nuss. Bioinformatics: Bayesian Analysis of Phylogeny. James Smith

(Biology) and Amit Jain.

Basic Seismic Utilities package. Paul Michaels (Geophysics) and Amit
Biologically Inspired Computing. Crowley Davis Research (private

company)

Design Patterns

Projects (contd.)

Clusmon. A comprehensive web-based cluster monitoring software. (Joey Mazzarelli, Computer Science senior) Remote Power Control. A cluster of smart power strips to enable remote hard power on/off, cascaded power on/off etc. (Brady Catherman, Computer Science junior) Parallel Shell. A more capable parallel shell for system administration. (Mason Vail, Computer Science graduate student)

Clusmon: Cluster Monitor

Remote Power Control

Cluster Statistics

1179 jobs since July, adding up to about 88000 CPU-hours. Average CPU temperatures: 77F at low load and 100F at full load. The A/C is set to 65F with tolerance of 4F. Hardware failures: Extremely low.
One disk drive failed right after installation. The memory for one node failed.
Only one unscheduled downtime in the last three months. The A/C
compressor was cycling more than the factory set limit. As a result, it shut itself off. The CPU temperatures still remained below 115F after several hours! (as the air flow was maintained) The cluster was shut down as a precaution. The solution was to simply set a higher tolerance (4 degrees instead of 2 degrees)
The experiences gained in this project were used to help Geophysics set-up a 10 processor cluster and Mathematics a 20 processor cluster.

Further Work

Integrate Beowulf clusters with Condor grids. Develop a complete catalogue of programs illustrating each design pattern in PVM and MPI. Continue to team with researchers to help get their code up and running on clusters.

Tested (SATA Hard Disks) Compatibility Lists for the HighPoint X4 Enclosure.
SATA I and SATA II Hard Disk Manufacturer Hitachi Hitachi Hitachi Hitachi Hitachi Hitachi Hitachi Hitachi Hitachi Maxtor Maxtor Maxtor Maxtor Maxtor Maxtor Maxtor Maxtor Maxtor Maxtor Maxtor Maxtor Maxtor Maxtor Samsung Samsung Samsung Seagate Seagate Seagate 8/2/2006 Model HDS728080PLA380 HDS728080PLA380 HDS722580VLSA80 HDS722512VLSA80 HDS722525VLSA80 HDT722525DLA380 HDS722525VLSA80 HDS724040KLSA80 HDS725050KLA360 6L200M0 7B300S0 6H500F0 6H500F0 6V320F0 6V250F0 6B250S0 7B250S0 7Y250M00654RA 6Y160M004573E 6Y160M004551A 6Y120M004530A 6Y080M0 6Y060M042501A SP4002H SP0812C HD160JJ ST360015AS ST380011AS ST380013AS Firmware PF20A50D PF2OA60A V32OA6MA V330A60A V360A60A V440A80A V360A6MA KFAOA20N K2AOA51A BANC1G10 BANC1B70 HA431C00 IHA421BMO VA101600 VA101600 BANC1B10 BANC19R0 YAR51EW0 YAR51GW0 YAR51EW0 YAR51BW0 YAR51EW0 YAR51VW0 QU100-04 SU100-25 WU100-33 3.01 3.05 3.15 Interface SATA II SATA II SATA I SATA I SATA I SATA II SATA I SATA I SATA II SATA I SATA I SATA II SATA II SATA II SATA II SATA I SATA I SATA I SATA I SATA I SATA I SATA I SATA I SATA I SATA II SATA II SATA I SATA I SATA I Capacity (GB) 80
SATA I and SATA II Hard Disk Seagate Seagate Seagate Seagate Seagate Seagate Seagate Seagate Seagate Seagate Seagate Seagate Seagate Seagate Seagate Seagate Seagate Seagate Seagate Seagate Seagate Seagate WD WD WD WD WD WD WD 8/2/2006 ST380819AS 380013AS 380023AS ST3120022AS ST3120023AS 3120026AS ST3160023AS ST3160827AS ST3160812AS 3160027AS 3160023AS ST3160827AS ST32508020AS ST3300832AS ST3300831AS ST3400832AS ST3400622AS ST3400632NS ST3500641AS ST3500641NS ST3750640AS ST3750640NS WD740GD WD740GD WD800GD WD1200JB WD1500ADFD WD1600JD WD2500JD 3 3.18 3.01 2.51 3.01 3.05 3.17 3.4 3.AKB 3 3.03 3.03 3.AAC 3.03 3.03 3.03 3.AAH 5.01 2.AAA/3.AAE 3.AEC 3.AAC 3.AAC 31.08F31 27.08D27 21.08U21 06.00R06 20.07P20 00HBB0 02.05D02 SATA I SATA I SATA I SATA I SATA I SATA I SATA I SATA I SATA II SATA I SATA I SATA I SATA II SATA I SATA II SATA I SATA I SATA I SATA II SATA II SATA II SATA II SATA I SATA I SATA I SATA I SATA I SATA I SATA I 250
SATA I and SATA II Hard Disk WD WD WD WD WD WD WD WD WD WD WD WD WD WD WD2500KS WD2500JD WD2500KS WD2500YD 2500JD 2500JD 2500JS WD2500JD WD3200SD WD3200JS WD3200KS WD4000YR WD5000KS WD5000YS 02.01C03 08.02D08 02.01C03 1.00E02 02.05D02 08.02D08 02.01c03 22HBB0 08.05J08 21.00M21 21.00M21 01.06A01 6.02E06 07.02E07 SATA II SATA I SATA II SATA II SATA I SATA I SATA II SATA I SATA I SATA II SATA II SATA I SATA II SATA II 500 500

8/2/2006