Empirical testing of Open Source Operating System Reliability
Anandeep Pannu & John Kew Open Source Software Lab (OSSL), Microsoft
http://port25.technet.com

Introduction

What is the OSS Lab? Motivation Experimental Setup Results So what does this all mean? Conclusion & future work
Technology Understanding Change Agent Cultural Understanding
Innovation Innovation Interoperability Interoperability Technology Research Technology Research

Motivation

Product groups want to know - how reliable is Linux? How does it compare? What can we do to assess reliability if the source code is available? Can the community process introduce enhancements that can be then be measured again?

Experimental Setup

Pick distributions of Linux Re-compile kernel with fault inducing modules Crash the system Analyze event logs Introduce enhancement to kernel Repeat
Results with standard kernel
Type of event Kernel panic Memory overwrite Boot disk failures User space program induced freezes Detected? No No No No

Power failures

No (for most hardware)
Results with enhanced kernel

(kdump + heartbeat)

Type of event Kernel panic Memory overwrite Boot disk failures User space program induced freezes Detected? Yes Yes No Yes
So what does this all mean?
Event logs are soooooo important! Little enhancements lead to giant advances A way for developers to test out if crashes are diagnosable (useful for someone else besides academics!) Open source systems will change the way empirical reliability research is done
Events co-related to environments to bugs to modules to developers Predict reliability using techniques for software code analysis?
Conclusion and future work
Complete the loop deploy in the field Use Linux 2.6.20 built in fault injection to obtain more data Start linking back errors to those damn developers! Larger scale study of open source software reliability
2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Experimental Setup Hardware

Dell Optiplex GX270

Intel Pentium IV 3GHz, 800MHz FSB 1 GB DDR SDRAM 400 MHz 80 GB 7200RPM Parallel ATA, ATAPI 48X DVD/CD-R/RW Intel PRO Gigabit Network 16550 Serial Port
Toshiba Portege 7200 CTe Series
Pentium III 650 Mhz E (256K L2 cache on die) 192MB PC100 SDRAM 8GB EINDE ATA-4 Interface Docking Station: 16550 Serial Port, 20x DVD/CD, Ethernet port

HP Proliant DL380

2 x Intel Xeon 3.2 Ghz processors with 2MB Cache 1 GB DDR2 666Mhz RAM 3 x 36GB 15K RPM, Ultra 320 80pin SCSI HD, Hot Plug, Smart Array, StorageWerks, RAID 1.44 Floppy, 24xCDROM Drive
Experimental Setup Software
Version Kernel Notes OS RHEL4 AS 2.6.9-42.0.3ELsmp Base install of RHEL 4-AS, with all current up2date patches. Server only. Base install of RHEL 4-AS, with all current up2date patches. Base install of Ubuntu 6.06 (Dapper) with current patches. Workstations only.

RHEL4 - WS

2.6.9-42.0.3EL

Ubuntu 6.06

2.6.15-27-386
Experimental Setup Data Collection

Logging Server

HTTP Status Logger Serial Console Logger Analysis Tools HTTP Test Reporting

Log Files

Test System Testing Dell Optiplex GX270 Scripts/ Modules/ OR Programs
Serial Cable Console Logging

Toshiba Portege 7200

HP Proliant DL380 Server
Experimental Setup Tests Performed
Six primary tests, and one control experiment:
NormalSystemStartup(Control) KernelPanicviaKernelmodule MemoryOverflowModule Crashme SimulatedPowerOutage RemovalofNonBootHardDrive RemovalofBootHardDrive

Data Collected

Scenario
NormalSystemStartup KernelPanicModule
Trivialdifferencesinlogs(PID,times,Mhz) Instantcrash.Noindicationoffailure exceptthroughtheabsenceof shutdown.
ModuleOverflowModule Instantcrash.Noindicationoffailureexceptthroughabsenceof shutdown.Attemptedwritesat0x219af3(kernelcode), 0x329af4(kerneldata),and0x3ccb78(1bytepastthekerneldata section) Crashme Power Nocrashes.Slowresponse.Usedseeds789,279,and666.No indicationoffailure exceptthroughabsenceofshutdown Noindicationoffailureexceptthroughabsenceofshutdown.
NonBootDriveRemoval Crashwithinafewseconds.DriveandATAPIerrorsinlog beforefull (CDROM) crash. BootDriveRemoval Nocrash,butunabletoshutdownproperly.Programswhichhad alreadybeenloadedintoRAMworked.Manyconsolemessages,but nolog (becausethedrivewasdisconnected).