Local Disk C:
Cluster Group Data Protector

2 HBAs

MSL5000

Robotics Tape Drives

SAN Library Quorum Disk Q:

Cluster

EVA Shared Disk E:

Data Protector

Shared Disk F:

Shared Disk G:

Additional Disks
The setup of clustered applications outside a cluster-aware Cell Manager can be considered as a subset and is therefore not described.

Installation

Support Matrices
Before installation, please check the latest support matrices for Data Protector and Microsoft Cluster Server at the following links: HP OpenView Storage Data Protector
http://www.hp.com/go/dataprotector http://www.microsoft.com/whdc/hcl
Products Designed for Microsoft Windows Windows Catalog and HCL

Software Removal

If a system in the cluster has the Data Protector software already installed, you need to uninstall it before the setup.

Cell Manager

Data Protector has a cluster-aware, fully automated installation of the Cell Manager. When Data Protector is installed on one single node, it is simultaneously installed to every other node. The installation succeeds with the following screen:
Figure 2. Successful Installation Status of Cell Manager Cluster
The installation procedure, which was started on TPC020, imported cluster node TPC021 and virtual node CLVIP16. After this, all physical and virtual nodes are known to Data Protector and can be used for configuration.

Client

The Data Protector client software cannot be push-installed to the nodes of a Microsoft Cluster Server. Therefore, the client software must be locally installed. The Data Protector client installation does not propagate to other cluster nodes. The installation procedure requires client installations on every cluster node. During installation, Data Protector will recognize the cluster and install the client in the cluster-aware mode. The cluster integration component is selected by default. The following figure shows the finished installation process with a message that the cluster node has not been imported:
Figure 3. Successful Installation Status of Cluster Client
After installing the client software on all cluster nodes, the complete cluster can be imported with the Client GUI and the Import Cluster feature.
Password Change of Data Protector User Account
If it is needed to change the user password after the installation, the following steps are required:
1. Set the cluster resources OBVS_VELOCIS and OBVS_MCRS offline. 2. Change on all cluster nodes the password of the Windows services Data Protector Inet and 3. Restart on all cluster nodes the Windows service Data Protector Inet. 4. Set the cluster resources OBVS_VELOCIS and OBVS_MCRS online again.

Data Protector CRS.

Debug and Log Files
Setup.exe (csetup.exe) automatically writes a detailed installation log to OB2DBG*.TXT. These log files contains complete information on setup except on the part where dialogs are being displayed (technical limitation of MSI itself). Details: <TEMP>\OB2DBG_<DID>__setup_<HOST><DEBUG_NO>.txt <DID> (debugging ID) is the process ID of the first process that accepts the debugging parameters. This ID is used as an ID for the debugging session. All further processes will use this ID. <HOST> is the name of the host where the trace file is created. <DEBUG_NO> is a number generated by Data Protector. The location of the <TEMP> directory is specified by the Windows TEMP environment variable. To examine the value of this variable run the set command in a DOS box.

Patches

Cluster patches must be locally installed. Its the same procedure as the Data Protector software installation. Patches can be installed on any node, which will be simultaneously distributed to any other node. Note: CORE and CS patches may require a reboot, which could result in a failover of the Cell Manager. Please check the patch description in advance.
It is the same procedure as for the Data Protector client software installation, which must be executed on every cluster node separately. Afterwards, the cluster must be re-imported.

Configuration

First, please check chapters Pre-Installation Requirement of File Share Resource and Post-Installation Check of Resource Dependencies for known issues. Then, follow the instructions in the HP OpenView Storage Data Protector 5.5 Installation and Licensing Guide. The following figure shows the two-node cluster CLNODE2 with belonging resources. Please note that Disk E: is destined to hold the shared Data Protector software and configuration. Disk F: and Disk G: are optional disks for future use in the same cluster group.
Figure 4. Cluster Administrator View of Data Protector Cluster Configuration
Automatic Restart of Backups
If a failover of the cluster-aware Data Protector Cell Manager occurs during backup, all running and pending backup sessions will fail. In the Data Protector GUI and in the backup specification, you can set one of the options that define automatic backup session restart at failover of Data Protector. Note: If the Cell Manager and another application are installed in the same cluster, its cluster critical resources need to be configured in the same cluster package or group as the application being backed up, in order to automatically restart failed backup sessions that failed due to a failover. Otherwise, the failed backup sessions must be restarted manually.

Load Balancing at Failover
If a failover causes a move of applications to that system where the Data Protector Cell Manager is running, it can result in a very high load on that system. Therefore, it might be necessary to abort running backup sessions in order to have enough system resources for the applications. It is possible to: Abort all running backup sessions. Abort specific running backup sessions. Inhibit the Data Protector cluster Cell Manager for a specific timeframe. The belonging feature is provided by the command line utility omniclus.exe.
Autoconfiguration of Backup Devices
The autoconfiguration wizard enables easy and automatic configuration of libraries in cluster environments. The devices are configured as virtual and floating drives. Floating drives are devices that are configured on a virtual host, using virtual hostnames. Floating drives should be configured for the backup of cluster-aware applications. This ensures that no matter on which node in the cluster the application is currently running, Data Protector always starts a media agent on that same node. Before executing Autoconfigure: Verify that drive hardware paths are identical on every node. Autoconfigure runs only on the active node without checking the inactive nodes. Therefore, drive hardware paths are only determined on the active node. During execution of Autoconfigure: Choose the appropriate virtual hostname. Select required devices. Enable option Automatically discover changed SCSI address, which could help in case of changed backup device paths (this is a well known problem in Windows environments, which can be detected by Data Protector). After execution of Autoconfigure: A failover test is imperative for checking that all paths are matching.
The next figures show an example how to autoconfigure a HP MSL5000 library. In the Scoping Pane, right-click Devices and click Autoconfigure Devices to open the wizard:
Figure 5. Device Autoconfiguration Wizard - Start
Select the client system by its virtual hostname, which will execute the autoconfiguration on the active node:
Figure 6. Device Autoconfiguration Wizard Client Systems
Please note that any device can be accessed from two paths, which is caused by two installed host bus adapters (HBA). Select the tape library with all required tape drives:

Figure 7. Device Autoconfiguration Wizard - Devices
Please enable wizard option Automatically discover changed SCSI address:
Figure 8. Device Autoconfiguration Wizard - Options
The following figure shows the final library robotics configuration. The robotics can be accessed via the virtual hostname and two different paths (multipath).
Figure 9. Final Library Configuration
The next figure shows the final configuration of library drive COMPAQ:SDLT320_1. The drive can be accessed via the virtual hostname and two different paths (multipath).
Figure 10. Final Drive Configuration

Busy Drive Handling

Libraries in clustered environments are managed by the active cluster node and the virtual hostname. If the active cluster node fails during a backup, the tape drive could not be unloaded due to an aborted and failing media agent. The next backup going to that drive will recognize an already loaded tape and fail. This can be prevented by setting an appropriate busy drive option. Note: It is highly recommended that libraries configured with virtual hostnames are capable to eject already loaded tapes. This can be achieved by setting the library option Busy drive handling.
Figure 11. Busy Drive Handling Library Control Option

Cluster Recovery

This chapter describes some common scenarios, which can be manually fixed and dont require an Automated System Recovery (ASR).

Cluster Disks

Shared disks use the disk signature to identify a disk and to map the real device to a physical disk resource. When a physical disk resource fails and is replaced, the signature of the newly formatted disk no longer matches the signature stored by the physical disk resource. Cluster and quorum disks can be recovered with the Microsoft Windows Server 2003 Resource Kit tool ClusterRecovery, which substitutes the newly created physical disk resource for the failed resource. The properties of the failed resource are automatically transferred to the new resource. Any dependencies on the old resource are changed to point to the new resource. Note: In case of a failed quorum disk, the cluster service must be started with parameter /fixquorum. The following two figures show a recovery of disk resource Disk F:. Before executing ClusterRecovery, the new disk resource must be visible to the same set of nodes as the old resource.

Figure 12. Server Cluster Recovery Utility Replace Physical Disk Resource
Figure 13. Server Cluster Recovery Utility Select Disk Resources
To complete the replacement, the new disk resource should be brought online and the drive letter should be changed with the disk management snap-in to match the old disk resource (this is necessary because applications typically reference files on the disk via a drive letter). If the new physical disk resource is successfully validated, the old physical disk resource should be deleted, as it no longer represents a real resource on the cluster. Once the cluster is configured with the new physical disk resource, the application data can be restored to the disk.
In case of problems, please check the file <File Share>\log\Server\cluster.log. Furthermore, debugs can be created:
1. Take the CRS service resource OBVS_MCRS offline. This will automatically stop the CRS service. 2. Edit the following lines within the trace file in the shared directory
(<File Share>\Config\Server\Options\trace): ranges: 1-99 postfix: DBG select: <virtual host name> 3. Take the CRS service resource OBVS_MCRS online again. This will automatically start the CRS service. Another possibility to start the CRS service in debug mode is to add the debug options (-debug 1-99 DBG) to the start parameters of the service resource OBVS_MCRS parameter properties. This is done within the Cluster Administrator GUI. The debugs can be found in the $DPHOME\tmp directory of the active node.
Automated System Recovery (ASR)
Automated System Recovery is a complex process and must be well organized. It is not sufficient to backup the environment. To be prepared for the real disaster, the disaster recovery process must be executed and verified in advance. Important issues are complete backups, appropriate network settings and correct drivers for mass storage controllers - particularly missing HBA drivers will prevent ASR from recognizing and recreating shared disks. Note: Only local shared storage (connected to cluster nodes via SCSI) is fully supported in cluster environments for ASR. Shared storage on disk arrays connected to cluster nodes via Fibre Channel (for example: EVA or XP disk arrays) is only supported if appropriate device drivers are provided during the initial phase of ASR recovery (by pressing F6). This enables Windows 2003 Setup to correctly detect shared storage located on disk arrays.

Cluster Backup

One single full backup session is a prerequisite for a successful ASR, which must include: All physical nodes including local disks. All virtual nodes including shared disks. The IDB virtual node, if Data Protector is configured as a cluster-aware application. Note: Perform a full client backup after each hardware, software or configuration change and update the ASR diskettes. This also applies to any network configuration changes, such as change of the IP address or DNS server.

Hardware Documentation

It is recommended to create a preparation template, as required for the Assisted Manual Disaster Recovery. It could be useful in case the ASR partly fails and e.g. shared disks must be manually created. Please check the HP OpenView Storage Data Protector 5.5 Administrators Guide Appendix A Further Information Windows Manual Disaster Recovery Preparation Template. Furthermore, the configuration of any cluster-aware tape device should be documented including hardware paths. This is valuable information for the local restore mode.

ASR Set

An ASR set is a collection of files stored on three (32-bit Windows) or four diskettes (64-bit Windows), required for: Proper reconfiguration of the replacement disk (disk partitioning and logical volume configuration). Automatic recovery of the original system configuration. The user data that was backed up during the full client backup. These files are stored on the Cell Manager as an ASR archive file in the shared directory <File Share>\Config\Server\dr\asr as well as on the backup medium. Note: Create the ASR set for the Cell Manager in advance, because you will not be able to obtain the ASR archive file after the disaster. ASR sets for other systems can be created using Cell Manager when a disaster occurs.

DHCP and DNS

For online ASR, DHCP must be additionally configured, though Microsoft cluster network interfaces must be configured with static IP addresses. DHCP must enable DNS Server, which resolves exactly all involved servers into correct full-qualified hostnames. E.g., online ASR will even fail if only the domain is different. If DHCP and DNS fail, the ASR offline restore mode is executed.

Restore Modes

Data Protector attempts automatic media management operations during ASR, which is called online mode. If the Cell Manager is inaccessible, the offline mode is executed. Note: The recovery of the cluster-aware Cell Manager is always offline. Online The online mode requires full access to the Cell Manager. Offline If the Cell Manager is inaccessible, the offline mode attempts to start the media agent including robotics support. Local If the online and the offline mode fail, Data Protector executes the local mode. All local devices are scanned and offered in an additional window. In this scenario, it is very helpful to verify the correct device and its hardware path with the latest hardware documentation.

Process Flow

The following example describes the process flow with all required steps for a cluster-aware Cell Manager configured with two nodes. It is assumed that the active node is completely lost including the shared disks, e.g. that a virus or bad program deleted all volumes with its data. In this case, ASR would restore the disk signatures and partition layouts of all cluster disks. Then, Data Protector would restore all critical volumes (quorum and Data Protector DPSHARE). After ASR has finished, noncritical volumes must be manually restored.
The following figure describes major steps of ASR:
Figure 14. Simplified ASR Process Flow
Boot from Windows Server 2003 CD Select ASR Mode Apply Drivers Insert ASR Diskette #1 Format of Target Disk (C:) Reboot Windows Server 2003 Installation Format of Critical Volumes

Data Protector ASR

DR wizard Insert all ASR diskettes

drstart.exe

omnidr.exe
Restore of Critical Data File Protection Catalog Filesystem and IDB Windows Configuration

Reboot

The next list describes the ASR steps in detail:
1. Make sure that you have the following available:
- The latest ASR floppy disks (as long as the cell manager is available, they can be created or updated with the disaster recovery wizard). - The latest backup media. - The original operating system installation CD. - If you have a mass storage controller and you are aware that the manufacturer has supplied a separate driver file for it (different from driver files available on the Setup CD), obtain the file (on a floppy disk). 2. Verify that the second node is switched off and does not access shared disk resources. 3. Insert the original operating system installation CD. 4. Restart the computer. 5. If you have a separate driver file as described in step 1, use the driver as part of Setup by pressing F6 when prompted. 6. Press F2 when prompted at the beginning of the text-only mode section of Setup. 7. If required, apply a separate driver file as described in step 1. 8. If prompted, insert the ASR floppy disk. 9. If prompted, confirm the delete and format partitions screen. 10. If prompted, re-apply a separate driver file.
11. If prompted, re-insert the ASR floppy disk. 12. ASR automatically reboots. 13. Remove the ASR floppy disk, which will later cause the DR process to stop and to offer additional 14. The ASR starts the preparation of the operating system installation, which takes a couple of 15. In the preparing installation step and immediately after installing devices and network, all cluster
options (Registry Editor, Command Window, Task Manager, Debug, Install Only). minutes.
disks are automatically formatted. If not, a problem with mass storage driver could be the reason as mentioned in step 1. 16. The DR wizard starts. Open at least one CMD-window. This enables file system access in case of problems. 17. If prompted, enter the drive letter and path of the SRD-file. 18. Follow the directions on the screen and insert all ASR floppy disks (which are copied to \WINDOWS\system32\OB2DR\bin). 19. Select wizard option Finish. 20. ASR starts Data Protector the OMNIDR utility (omnidr.exe srd recovery.srd), which executes the following steps: - Start INETD daemon. - Check online restore mode, which will fail in this scenario due the DR of the Cell Manager. - Attempt offline restore mode. It could fail in case of an outdated SCSITAB or mismatch of robotics and tape drivers (e.g. "scsi5:1:0:2" instead of "Changer0:1:0:2"). The outdated SCSITAB problem and its solution are described in chapter ASR and Outdated SCSITAB and the driver problem in chapter ASR and Mismatch of Robotics and Tape Drivers. - If offline restore mode fails, local restore mode is started. A device menu will prompt you to select a device with the loaded tape media. Please make sure that the media is loaded before the appropriate device is selected. Otherwise, ASR will fail and finish. - Data Protector starts restore sessions one after the other for the Windows file protection catalog, the operating system volume (C:) in parallel with the Data Protector shared volume (DPSHARE), and the Windows configuration database (registry). 21. ASR reboots automatically. After the successful ASR, restore all non-critical volumes as described in the HP OpenView Storage Data Protector 5.5 Administrators Guide.

Per default, Data Protector creates ASR debug and log files in the following directories: \WINDOWS\system32\OB2DR\tmp\debug.log \WINDOWS\system32\OB2DR\bin\OB2DBG*.txt
Additional Support Information
In the past, many questions were asked regarding the support of network card teaming and RAID type of disk arrays. Network card teaming is supported and Data Protector is able to determine the correct IP address, which is saved in the SRD-file. RAID technology is supported but could cause more effort during disaster recovery due to additional steps with disk array belonging tools. If the disk array is lost, the same volume configuration must be re-established for a successful ASR.

Known Issues

Pre-Installation Requirement of File Share Resource
File Share Dependencies The <File Share> resource, where Data Protector has to be installed, must have the following resources set among the <File Share> dependencies (as described in the HP OpenView Storage Data Protector 5.5 Administrators Guide chapter Installing Data Protector on Microsoft Cluster Server): IP Address Network Name Physical Disk Before installation, check the file share resource for the following dependencies:
Figure 15. Dependencies of File Share Resource
File Share Permissions Microsoft Windows Server 2003 gives only read access to everyone. Therefore, configuration file omni_info cannot be created and installation directory <File Share>\Config\ Server\install is empty. At this stage Data Protector needs:
1. To be uninstalled. 2. File share permissions to be fixed. 3. To be reinstalled.
Workaround: To avoid this problem before installing Data Protector in a MS Cluster, a Data Protector cluster group containing the future DP folder as file share resource type needs to be created. Then all operations to everyone (or at least the accounts that will write in this folder or subfolders) must be granted. It is planned to fix this problem in the future. Please check the latest patches.
The following figure shows the corrected permissions for file share resource DPSHARE:
Figure 16. Permissions for DPHSHARE

Post-Installation Check of Resource Dependencies
After installation, check the following resources for correct dependencies:
Figure 17. Dependencies of OBVS_MCRS
Figure 18. Dependencies of OBVS_VELOCIS
Figure 19. Dependencies of OmniBack Share
Figure 20. Dependencies of Network Name
Tape Devices and Failover
Tape devices connected to cluster nodes and its loaded media require additional attention after failover. Note: If a failover during backup activity occurs, the Data Protector may not be able to properly abort the session. Data may not be completely written which results in the corruption of the medium. It is highly recommended to restart the belonging backup session.

Cluster Database Restore

Two major problems are known:
1. The restore session of the cluster database fails because the ASR process restarts the cluster
services during the session. When DP starts the cluster database restore, it calls the belonging Windows API. The API stops the cluster services and with it DP session connections, before the restore session has finished. 2. The cluster database restore of a clustered Cell Manager may result in a failover. Note: Before starting the restore of a cluster database, you should stop the cluster service on all inactive nodes. It is planned to fix this problems in the future. Please check the latest release notes and patches.

ASR and Windows RSM

The Windows Removable Storage Manager (RSM) is activated during ASR. RSM is a service used for managing removable media (such as tapes and discs) and storage devices (libraries). With RSM and SAN shared libraries, there are known issues. RSM automatically scans system buses after reboot to determine if there are any removable storage devices available for Windows to manage. This can interrupt backup and restores by doing a poll of connected devices causing a SCSI reset on shared buses. Additionally, there is a conflict of ownership for the removable storage devices. If the RSM service is running, it captures library devices and maps them to Windows. The devices will be no longer available for Data Protector. Note: The RSM service should be manually disabled after the ASR process has finished (after the final reboot). A solution will be provided in the near future. Please check the latest AUTODR patch.

ASR and Local Tape Devices
If the Data Protector Cell Manager together with local tape devices is installed cluster-aware, the ASR process cannot access the local tape devices by its virtual hostnames. ASR enters the local restore mode, starts a device autoconfiguration and sets the tape device hostname temporarily to machinename, which does not match the name as described in the SRD-file recovery.srd. Change in the SRD-file the system name specified with mahost file to machinename in order to avoid the local restore mode. Please check in the HP OpenView Storage Data Protector 5.5 Administrators Guide chapter Advanced Recovery Tasks how to edit the SRD-file. If the SRD-file is not changed, the autoconfiguration process will determine any available tape devices and offer it to the disaster recovery wizard. The devices and their hardware paths are displayed. You must insert a tape into the device and choose the correct path in the wizard. This can be quite complicated if the library is SAN-attached and drives are multiple displayed. In this case, the latest
hardware documentation is required to determine the correct path. If a wrong device is chosen, ASR will fail and the process must be repeated.

ASR and Outdated SCSITAB

If you are using a locally attached device for ASR, test if its supported. E.g., if the original DP 5.5 released SCSITAB-file was updated with information for the library used for the disaster recovery process, it must be copied to the first ASR disk. Please check in the HP OpenView Storage Data Protector 5.5 Administrators Guide chapter Disaster Recovery / Automated System Recovery / Preparation.
ASR and Mismatch of Robotics and Tape Drivers
Due to historical issues of Data Protector, the OMNIDR utility (omnidr.exe) utility always disables tape/robotics drivers, which are enabled per default by ASR. Example: Tape1:0:0:0 becomes scsi1:0:0:0. The same happens to the robotics devices. This is exactly what was necessary to be done in former Data Protector releases, because all devices were typically installed as scsi#:#:#:# and not Tape#:#:#:#. The mismatch driver problem could be fixed in the SRD-File by replacing all Changer and Tape entries (-devioctl and -devaddr) with the appropriate SCSI-paths:
1. After the DR wizard starts, please open at least one CMD-window as described in step 16 of

chapter Process Flow.

2. Execute the OMNIDR utility: omnidr.exe srd recovery.srd. This will disable all tape/robotics
drivers (must run before devbra.exe is executed). 3. If it fails, determine the appropriate SCSI-paths by executing devbra.exe -dev in the CMDwindow. 4. Modify the SRD-File as described above. 5. Restart the OMNIDR utility: omnidr.exe srd recovery.srd.

ASR and Shared Disks

This chapter covers the EVA only. Other disk arrays might have different procedures. Note: Before executing ASR, its highly recommended to start with clean and unpartitioned virtual disks. If not, the ASR process could skip the discs due to a function that avoids overwriting existing data. The virtual discs would have to be manually created and recovered afterwards. If the problem with the mass storage controller, as described in chapter Automated System Recovery (ASR), cannot be fixed and shared disks are not correctly recognized, a workaround with manual disk partitioning and formatting could be executed. In this case, execute steps as described in chapter Process Flow. Then, follow this procedure:

1. Select wizard option Install Only. 2. Follow the directions on the screen and insert all ASR floppy disks (which are copied to
\WINDOWS\system32\OB2DR\bin). 3. Determine the cluster disk numbers (Disk #) in the diskinfo section of the SRD-File. First, search for the appropriate drive letter in the parameter -letter and the belonging signature in -volume: -volume 1141165761 -number 2 -letter Q Then, search with the just determined signature in -layout for the belonging disk number in the disk section: -disk 2 -addr 0 -sizelow 0 -sizehigh 0 -descr "Compaq Secure Path Disk" -geo 12 -cyl 261 -tpc
255 -spt 63 -bps 512 -layout 1141165761 -partcount 4 In this example, disk 2 would be the searched disk. 4. In the CMD-window, execute: \WINDOWS\system32\bin\diskpart.exe. - List all disks, e.g.: DISKPART> list disk - Select the correct disk with the determined disk number, e.g.: DISKPART> select disk 1 - Create a partition, e.g.: DISKPART> create partition primary. - Assign a drive letter, e.g.: assign letter=Q - Repeat the steps for all remaining cluster disks - create partitions and assign drive letters. - Finish diskpart. 5. Format all partitioned disks of step 5 from the CMD-window, e.g. C:\WINDOWS\system32> format Q: /fs:ntfs 6. In the CMD-window, execute \WINDOWS\system32\bin\omnidr.exe srd recovery.srd. This will start multiple restore sessions for Windows file protection catalog, critical data including IDB, and Windows configuration. 7. After successful restore, select the Abort button. This will finish the Install Only mode. 8. ASR reboots automatically. 9. Install Microsoft Resource Kit Tools. 10. Execute ClusterRecovery and replace all bad disk resources. 11. Start second cluster node. 12. Delete all bad disk resources. Note: The DR wizard option WinDisk, which would start the disk manager, cannot be used during ASR.
Microsoft Windows Server 2003 Resource Kit Tools
The resource kit contains tools, which allows fixing cluster problems. E.g., ClusterRecovery helps to recreate shared disks and resource checkpoints. Please check the Microsoft documentation for use cases and appropriate tools. Microsoft Windows Server 2003 Resource Kit Tools

http://www.microsoft.com/windowsserver2003/downloads/tools

Summary

This white paper describes the Microsoft Windows Server 2003 Cluster Server integration issues of HP OpenView Storage Data Protector 5.5. Installation, software maintenance (patches), configuration and cluster recovery issues are covered with examples. Additionally, known issues are documented for avoiding common problem cases. The described limitations and notes should be very well considered.

For more information

HP OpenView Storage Data Protector
http://www.hp.com/go/dataprotector
HP OpenView Storage Data Protector 5.5 Guides
http://ovweb.external.hp.com/lpe/doc_serv
HP OpenView Storage Data Protector v5.5 Support Matrices
http://www.openview.hp.com/products/datapro/spec_0001.html
Windows Server 2003 Microsoft TechNet
http://www.microsoft.com/windowsserver2003 http://www.microsoft.com/technet
Guide to Creating and Configuring a Server Cluster under Windows Server 2003 Products Designed for Microsoft Windows Windows Catalog and HCL
http://www.microsoft.com/whdc/hcl
http://www.microsoft.com/technet/prodtechnol/windowsserver2003/technologies/clustering
Microsoft Windows Server 2003 Cluster Documentation
http://www.microsoft.com/resources/documentation/WindowsServ/2003/standard/proddocs/enus/Default.asp?url=/resources/documentation/WindowsServ/2003/standard/proddocs/en-us/win_cluster.asp
2003 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Itanium is a trademark or registered trademark of Intel Corporation in the U.S. and other countries and is used under license. XXXX-XXXXEN, 07/2003

HP OpenView Storage Data Protector 5.5 Advanced Backup to Disk Performance White Paper
Executive Summary.... 3 Objective and Target Audience.... 4 Introducing HP OpenView Storage Data Protector 5.5.... 5 File Library.... 5 Object Copy..... 5 Tape Drives..... 6 Basics of Backup Performance.... 8 Disk to Disk to Tape (D2D2T) Data Protection Architecture... 10 Test Data..... 12 HPCreateData.... 12 Typical Files..... 12 Small Files..... 13 HPReadData..... 14 Configuration.... 15 HP ProLiant DL380 G4 Server.... 15 HP OpenView Storage Data Protector 5.5... 15 Internal Database (IDB) Logging Level... 15 File Library..... 16 File System Tree Walk..... 18 HP StorageWorks SDLT 320 Tape Drive... 18 HP StorageWorks Ultrium 460 Tape Drive... 19 HP StorageWorks Ultrium 960 Tape Drive... 20 HP StorageWorks Modular Smart Array 1000 (MSA1000)... 21 HP StorageWorks Modular Smart Array 1500 (MSA1500)... 22 RAID 0 and 1 Logical Array.... 22 RAID 0 and 4 Logical Arrays... 23 RAID 1 and 1 Logical Array.... 24 RAID 1+0 and 3 Logical Arrays.... 25 RAID 5 and 2 Logical Arrays... 26 Test Data..... 27 Library and Tape Tools.... 27
Results..... 29 Tape Drive Write Test.... 29 MSA1000 Write Test.... 29 MSA1500 Write Test.... 29 MSA1000 Read Test..... 30 MSA1500 Read Test..... 30 Backup to NULL Device.... 31 Backup to Tape.... 31 Advanced Backup to Disk.... 33 MSA1000 to File Library (MSA1500)... 33 File Library (MSA1500) to Tape.... 33 Restore..... 34 File Library (MSA1500) to MSA1000... 34 Tape to MSA1000.... 34 Observations..... 35 HP ProLiant DL380 G4 Server.... 35 Tape Drives and LTT.... 35 MSA Read and Write Tests.... 36 MSA1000.... 36 MSA1500.... 36 Backup to NULL Device.... 37 Backup to Tape.... 37 Advanced Backup to Disk.... 38 Restore..... 40 Tuning Recommendations.... 41 General..... 41 Fragmentation of File Library File Systems.... 42 Summary and Conclusions.... 45 For more information.... 46

Executive Summary

This white paper provides performance-related information for HP OpenView Storage Data Protector 5.5 and the Advanced Backup to Disk feature. This white paper covers HP ProLiant Windows 2003 server connected to HP StorageWorks Modular Smart Array (MSA) disk arrays and different tape drive technologies (LTO and SDLT). The proof points are all Windows-based for simplification of equipment needs, but the lessons will still hold good for heterogeneous environments. As a result of these tests, several recommendations and rules of thumb have emerged: HP OpenView Storage Data Protector tuning can help to improve the performance, e.g. by modifying tape block and file depot sizes. Please check chapter Tuning Recommendations. HP StorageWorks Ultrium 960 tape drives are best utilized with a block size of 256 KB as described in Getting the most performance from your HP StorageWorks Ultrium 960 tape drive white paper (downloadable from http://h18006.www1.hp.com/storage/tapewhitepapers.html). A single high-performance tape drive (Ultrium 960) causes less CPU load than multiple slower tape drives (e.g. 2 x SDLT 320) at a comparable or better transfer rate. For entry-level JBOD and low-end disk array usage, it is important to understand what your disk subsystem is capable of delivering. This can be done using HP performance assessment tools (downloadable from http://www.hp.com/support/pat). The performance tools are also embedded within the HP industry-leading Library and Tape Tools diagnostics (downloadable from http://www.hp.com/support/tapetools). The configuration of disk arrays can have a remarkable impact on the backup and restore performance. Important parameters are the configured number of logical arrays, logical volumes and type of RAID levels. The configuration of logical arrays and volumes should reflect the internal disk array layout. E.g., disk arrays with 2 SCSI buses could perform best by creating 2 logical arrays with one logical volume each. RAID levels must be chosen carefully. RAID 0 provides the best performance but should not be considered due to missing fault tolerance. Therefore, it is recommended to configure RAID 5, which fits best for staging areas. It is space and cost efficient and provides a good read performance. Serial ATA (SATA) disks used as secondary disk storage arrays can have slower rotational speeds than their SCSI counterparts (for example, the HP StorageWorks Modular Smart Array SATA drives are only 7.2K rpm) and although they are high capacity (250 GB), their performance reflects their pricing. Therefore, backing up to tape from a staged (secondary) disk array can be slower than backing up directly from the primary storage to tape in some circumstances. Ironically, tape is now faster than disk. Disk staging is useful, however, for gathering several small files into a single object, or backing up slow networked hosts before the data is sent to tape. Both small files and slow hosts can cause very slow backups. The restore of many small files (22 million in this setup) could cause serious file system bottlenecks. Data Protector must wait until Windows and the belonging NTFS responds. Finally, the test environment with the HP ProLiant Server DL380 G4 and MSA1000/1500 disk arrays is capable to manage data with low CPU utilization. The typical file backup (from MSA100) directly to the Ultrium 960 tape drive showed a high transfer rate (91,97 MB/s or 323,33 GB/h) together with a low average CPU load (11%). The comparable disk backup (from MSA1500) to tape still showed an acceptable transfer rate (43,37 MB/s or 152,42 GB/h) together with a low average CPU load (9%).

Objective and Target Audience
The main objective of this white paper is to educate and inform users of the HP OpenView Storage Data Protector 5.5 Advanced Backup to Disk feature about what levels of performance are achievable in different backup scenarios. The emphasis is in showing what is typical and not what best-case scenarios are. This white paper highlights where the current performance bottlenecks are and how these might be overcome. The target audience for this white paper are system integrators and solution architects and indeed anyone involved in getting the best backup performance out of their infrastructure investments. Throughout the white paper the HP StorageWorks Ultrium may also be referred to as HP LTO.
Introducing HP OpenView Storage Data Protector 5.5
HP OpenView Storage Data Protector (DP) is software that manages backup and recovery from both disks and tapes, delivering maximum data protection while providing continuous 24x7 business operations. The software is designed to simplify and to centralize backup and recovery operations by integrating a variety of techniques to eliminate backup windows. These range from on-line backup, open file backup, and instant recovery or zero-downtime backups. HP OpenView Storage Data Protector simplifies the use of complex backup and recovery procedures with the fastest installation, automated routine tasks, and easy-to-use features. The ideal solution to reduce IT costs and complexity while remaining reliable and scalable to grow from single server environments to the largest distributed enterprise infrastructures, providing broad compatibility of operating systems, applications, drives, libraries, and disk arrays. The Advanced Backup to Disk feature improves the backup process with continuous backup of transaction log files, backup of slow clients without multiplexing, easy resource access and sharing, plus backup in tape-less branch offices, while offering fast and easy configuration and licensing. Furthermore, it allows single file restore directly from disk or tape. The following subchapters File Library and Object Copy give a short overview about the features required for Advanced Backup to Disk. For further information on Advanced Backup to Disk, please refer to the Disk-Assisted Backup White Paper (downloadable from http://www.openview.hp.com/sso/searchdocs?prod=DATAPRO&ct=TWP).

File Library

DP provides a device type called File Library. It focuses on low cost disk arrays, especially the current HP SATA disk array, which is positioned mainly as a backup device. In case the file library is running out of free space, new backup capacity can be assigned automatically. The file library is configured by defining mount points where DP will create its media and optionally, the number of simultaneous writers that will be used. Then, DP can utilize this just like any other physical backup device and will auto-create the media files on the fly as required. This capability is called Advanced Backup to Disk.

Transfer Size Transfer size is the overall total size of the SCSI transfer within a single SCSI command. In some operating systems, there is limit set on this. For example, in Microsoft Windows the default transfer size is 64 KB and to increase the overall transfer size above 64 KB in Windows, a registry entry called MaximumSGList (associated with the HBA) must be changed. Many modern HBAs already install their drivers with this registry value set appropriately. Check the registry entries. Fragmentation The more fragmented the files are on disk, the more random will be the disk access method, hence the backup will take longer. If your system does have a de-fragmentation utility, it is advisable to run it before full backups or on a regular scheduled basis to ensure files are largely contiguously arranged on disk. SAN Inter Switch Links In the switched fabric, SAN Inter Switch Links (ISLs) ensure that SAN connections have sufficient bandwidth to support the backup traffic going through them. Trunk multiple ISLs together where possible. 2 Gigabit Fibre Channel Expect no more than 180 MB/s maximum from a single 2 GB/s Fibre Channel connection. SCSI Burst Rate Beware of disk drives quoted as Ultra320; Ultra 320 refers to the burst rate not the sustained rate. The typical sustained rate from a 15K rpm SCSI Ultra320 disk drive is around 80 MB/s for raw sequential I/Os (that is, without file system read overhead). SATA Disks These types of disks are lower cost, lower performance, and lower reliability than the SCSI disk previously listed but are useful for staging backups because they offer high capacity. A typical 7200K rpm SATA disk drive has lower seek times than an equivalent SCSI disk drive, a burst rate of around 150 MB/s, and a sustained transfer rate of around 50 MB/s.
Disk to Disk to Tape (D2D2T) Data Protection Architecture
With the high capacity and lower cost offered by SATA and Fibre Channel ATA (FATA) disk technologies, many customers are now considering implementing backup to low cost disk arrays before backup to tape. The use of secondary disk arrays for backup is best suited to environments where: The business dictates rapid single file restore capabilities (seconds to minutes). It is generally quicker to restore from a secondary disk subsystem than it is from tape (minutes to hours). However, the secondary storage array is not infinite in capacity and only the most recent backups may still reside on the disk array. In addition, disk-to-disk backup is no substitute for offsite media. The hosts can only supply data at a relatively modest rate (1020 MB/s). The backup image can be gradually produced on the secondary array without having to interleave multiple streams to tape and without the need for a large powerful dedicated backup server. Small file backup to tape has always been a performance limiter. With backup to disk, a complete backup image of small files can be constructed and then passed to tape at much higher transfer rates than if the small files were transferred directly to tape.

Table 2. Comparing SATA and SCSI Disks

Feature

Mean time to failure (MTTF) Burst transfer rate RPM Queuing

SATA Disks

500,000 hrs @ 20% duty cycle 150 MB/s 7.2K Non-Tagged Serial execution 250 GB 1 year

SCSI Disks

1,200,000 hrs @ 100% duty cycle 320 MB/s 10K or 15K Tagged Optimized seeks Better performance 146 GB 3 years

Capacity Warranty

While the SCSI disk MTTF of 1,200,000 looks high, this figure decreases when there are many disk drives bound together in an array. By comparison, Ultrium 460 tape drives have a MTTF of 250,000 at 100% duty cycle but are used in smaller volumes within an automated tape library. Therefore, do not assume that disk is automatically more reliable than tape. The key point is D2D2T has its place and with proper administration can improve the data protection process but it is not a replacement for tape. Tape is still the foundation of a robust data protection strategy. HP OpenView Storage Data Protector 5.5 provides a comprehensive implementation of D2D2T called Advanced Backup to Disk. The following example shows a scenario with a single central backup server and multiple network connected clients: The first step is to backup clients over the network to a central backup server and its staging area. The second step is to copy the consolidated backups from the staging area to tape. Typically the central backup and restore environment is designed to provide high speed backup from disk to tape. If a restore is required, data can be accessed either from disk (if still available) or from tape over the network.
Figure 4. DP 5.5 Advanced Backup to Disk Example
Data Protector 5.5 - Backup to Disk Disk Staging Example
Client 1. Backup client to disk (File Library) Backup server Client Staging Area Tape 2. Backup disk to tape (object copy)
Client 3a. Fast restore from disk (if data still available)
3b. Direct restore from tape
Use cases Continuous backup of transaction log files (avoids tape drives being in start/stop mode) Backup of slow clients without multiplexing Tape-less branch office backup Faster small file backup Resource sharing On the other hand, database backups are typically not first candidates for a D2D2T backup solution. The direct backup of large database files to tape can usually be done very efficiently, utilizing the full performance of the tape drive.

Test Data

In the following proof points, two different file systems were created to cover large file servers and typical clients, so that the results shown are realistically achievable in similar situations:
1. File server data with millions of small files 2. Typical client data with fewer files and a broad range of size (KB/MB)

HPCreateData

The datasets were developed using the HPCreateData PAT utility, which is downloadable from http://www.hp.com/support/pat. It generates different file sizes with different data contents (fixed, random, up to 4:1 compression ratio) and different distribution (file-based, MB-based).

Typical Files

The typical file system is created with file sizes between 64 KB and 64 MB and the compressibility of the data 2:1. The utility creates an equal distribution of files in each directory. Finally, the file system contains 49.8 GB with 4,389 files in 20 folders.
Figure 5. HPCreateData for Typical Files

Small Files

The small file system is created with file sizes between 4 KB and16 KB and the compressibility of the data 2:1. The utility creates an equal distribution of files in each directory. Finally, the file system contains 49.2 GB with 5,535,750 files in 7,380 folders. The created files have a name with maximal 16 characters for avoiding corner cases.
Figure 6. HPCreateData for Small Files

HPReadData

The HPReadData PAT utility is useful in assessing the rate at which your disk subsystem can supply data, and this is ultimately what will limit the backup performance. It simulates the way DP read files. A single instance of HPReadData can read eight streams simultaneously from your array. To read more than eight streams, initiate multiple instances of HPReadData. HPReadData is available for Windows, HP-UX, Solaris, and Linux. It can be downloaded free from http://www.hp.com/support/pat. The performance tools are also embedded within the HP industry-leading Library and Tape Tools diagnostics which are downloadable from http://www.hp.com/support/tapetools.
Figure 7. HPReadData for Determination of Read Performance
The above screenshot shows HPReadData reading one single LUN in a manner similar to the way a backup application will read files. We can see that the maximum read rate from this configuration is 57 MB/s, so we cannot expect any higher back-up transfer performance to tape than this figure.

Configuration

HP ProLiant DL380 G4 Server
The HP ProLiant DL380 G4 server features 2 x 3.2 GHz processors with 4 GB of RAM, 1 x 36 GB local disk, and Windows Server 2003. All file systems are configured as NTFS.
HP OpenView Storage Data Protector 5.5
HP OpenView Storage Data Protector 5.5 is configured with default values if not specified in the following subchapters. When more than one LUN is backed up, the data is multiplexed with the described concurrency parameter (see result tables). One NULL device is created for determining the max. read rate. This can be achieved by specifying nul in the SCSI address field of the device drive window.
Figure 8. Data Protector NULL Device
Internal Database (IDB) Logging Level The IDB logging level determines how much detailed information about backed up files and directories (name and versions) is written to the IDB. The tests are executed with option value Log Files, which is appropriate for many customer environments. With this level, directories and files can be browsed before restoring. Data Protector can also fast position on the tape when restoring a specific file or directory. The information written to the IDB does not occupy much space (as with Log All), since not all the file details (file attributes) are logged to the database. Please note that with Log Files a filename is only logged once and basically during the initial backup which requires more CPU and disk resources than the following backups, when the filename is already known.
File Library The HP OpenView Storage Data Protector Advanced Backup to Disk license (terabyte-based) enables the MSA1500 to be configured as a file library, so the initial backup from MSA1000 to MSA1500 takes place as a normal backup job with the file library being specified as the destination. The file library can be configured in much the same way as a tape device, with block size, segment count, disk buffers, and so on. It is important to realize that the backup file created on the MSA1500 is for all intent and purpose a tape image file. The benefit of this will be seen later when examining small file backup performance. Using HP OpenView Storage Data Protector Advanced Backup to Disk, the migration of the backup on the MSA1500 to tape is executed by means of a copy style function. It can be interactive (manual) or scheduled to occur at a certain time or even directly after the initial backup to the MSA1500 has completed. Two file libraries (FileLibrary, FileLibrary10GB) are created with different maximum file depot sizes (10, 50 GB).

Figure 9. File Library Configuration Directories and Drives
Figure 10. File Library Configuration Maximum File Depot Size 10 GB (50 GB Default)
Only for the Ultrium 960 tape drives, the block sizes of file library drives are configured with 256 KB (64 KB default), as described in Getting the most performance from your HP StorageWorks Ultrium 960 tape drive white paper (http://h18006.www1.hp.com/storage/tapewhitepapers.html).
Figure 11. File Library Configuration Drive Block Size 256 KB for Ultrium 960 (64 KB Default)
Note: For avoiding resource-intensive block repacking during backup from file library to tape drives, it is recommended to configure same block sizes for file library drives and the tape drives. E.g. Ultrium 960 tape drives perform best with the block size of 256 KB which should be also configured for the belonging file library drives.
File System Tree Walk During runtime, Data Protector creates backup statistics by a first tree walk, which briefly scans the files selected for the backup and calculates its size, so that the progress (percentage done) can be calculated. In a second tree walk, the data is written to the backup device. Note: If millions of small files are backed up, the first tree walk could take considerable time and increase the overall runtime. The tree walk can be disabled by setting NoTreeWalk=1 in the configuration file <Data_Protector_home>\omnirc.
HP StorageWorks SDLT 320 Tape Drive
The SLDT 320 tape drives are configured with default values.
Figure 12. HP StorageWorks SDLT 320 Tape Drive Test Environment
Advanced Backup to Disk SDLT320 Tape Drives
Primary Disk Array MSA1000 with 14 x 146GB/10K disks configured as RAID1+0, 8 x 100GB LUNs configured Secondary SATA-based Disk Array MSA1500 with 12 x 250GB/7.2K disks
Backup to Secondary Array 2 Gigabit Fibre Channel Backup to Tape from Secondary Array
HP ProLiant DL380G4 3.2 GHz, 4GB RAM, 2 CPUs
HP StorageWorks SDLT320 Tape Drives

Ultra2 SCSI

March 2005
Data Protector 5.5 - Advanced Backup to Disk Test Plan
HP StorageWorks Ultrium 460 Tape Drive
The Ultrium 460 tape drive is configured with default values.
Figure 13. HP StorageWorks Ultrium 460 Tape Drive Test Environment

Figure 20. MSA1500 Logical Array Configuration RAID 0 and 4 Logical Arrays
Figure 21. MSA1500 Logical Drive Configuration RAID 0 and 4 Logical Arrays
RAID 1 and 1 Logical Array The disk enclosure is configured as 1 logical disk array with 2 disks.
Figure 22. MSA1500 Logical Array Configuration RAID 1 and 1 Logical Array
1 LUN protected by RAID 1+0 is created. Due to having only 2 disks assigned, this is equivalent to RAID 1 (no striping).
Figure 23. MSA1500 Logical Drive Configuration RAID 1 and 1 Logical Array
RAID 1+0 and 3 Logical Arrays The disk enclosure is configured as 3 logical disk arrays with 4 disks each.
Figure 24. MSA1500 Logical Array Configuration RAID 1+0 and 3 Logical Arrays
3 LUNs protected by RAID 1+0 are created (only LUNs 1-2 will be utilized for tests) as shown in the next figure.
Figure 25. MSA1500 Logical Drive Configuration RAID 1+0 and 3 Logical Arrays
RAID 5 and 2 Logical Arrays The disk enclosure is configured as 2 logical disk arrays with 6 disks each.
Figure 26. MSA1500 Logical Array Configuration RAID 5 and 2 Logical Arrays
2 LUNs protected by RAID 5 are created as shown in the next figure.
Figure 27. MSA1500 Logical Drive Configuration RAID 5 and 2 Logical Arrays
Each LUN is loaded with 49.8 GB typical or 49.2 GB small files as previously described in chapter Test Data.

Library and Tape Tools

The tape drive write tests are executed with the HP industry-leading Library and Tape Tools diagnostics (downloadable from http://www.hp.com/support/tapetools). The tool is configured to create: Zeros with 64 KB block size 2:1 compressible data with 64 KB block size 2:1 compressible data with 256 KB block size
Figure 28. Library and Tape Tools Zeros with 64 KB Block Size
Figure 29. Library and Tape Tools 2:1 compressible data with 64 KB block size
Figure 30. Library and Tape Tools 2:1 compressible data with 256 KB Block Size

Results

Tape Drive Write Test
Table 3. Tape Drive Write Test
HP StorageWorks Ultrium 960 Transfer Rate (MB/s)
HP StorageWorks Ultrium 460 Transfer Rate (MB/s)
HP StorageWorks SDLT 320 Transfer Rate (MB/s)
Zeros, 64 KB Compr. 2:1, 64 KB Compr. 2:1, 256 KB

MSA1000 Write Test

Table 4. MSA1000 Write Test
RAID 1+0 and 1 Logical Array LUN #1 with Typical Files RAID 1+0 and 1 Logical Array LUN #1 with Small Files
MSA1000 Transfer Rate (MB/s)

76,53 14,22

MSA1500 Write Test
Table 5. MSA1500 Write Test
RAID 0 and 1 Logical Array LUN #1 (12 Phys. Disks) RAID 0 and 4 Logical Arrays LUN #1 (3 Phys. Disks) RAID 1 and 1 Logical Array LUN #1 (2 Phys. Disks) RAID 1+0 and 3 Logical Arrays LUN #1 (4 Phys. Disks) RAID 5 and 2 Logical Arrays LUN #1 (6 Phys. Disks) RAID 5 and 2 Logical Arrays LUN #1/2 (2 * 6 Phys. Disks)

MSA1500 Transfer Rate (MB/s)

MSA1000 Read Test

Table 6. MSA1000 Read Test
Typical Files LUN #1 Typical Files LUN #1-4 Small Files LUN #1 Small Files LUN #1-4

45,70 93,58 8,12 14,25

MSA1500 Read Test
Table 7. MSA1500 Read Test
RAID 0 and 1 Logical Array LUN #1 (12 Phys. Disks) RAID 0 and 1 Logical Array LUN #1-4 (12 Phys. Disks) RAID 0 and 4 Logical Arrays LUN #1 (3 Phys. Disks) RAID 0 and 4 Logical Arrays LUN #1-4 (12 Phys. Disks) RAID 1 and 1 Logical Array LUN #1 (2 Phys. Disks) RAID 1+0 and 3 Logical Arrays LUN #1 (4 Phys. Disks) RAID 1+0 and 3 Logical Arrays LUN #1/2 (8 Phys. Disks) RAID 5 and 2 Logical Arrays LUN #1 (6 Phys. Disks) RAID 5 and 2 Logical Arrays LUN #1/2 (2 * 6 Phys. Disks)

Backup to NULL Device

Table 8. Backup to NULL Device
NULL Device Transfer Rate (MB/s)
45,57 67,34 92,80 7,35 13,99 14,23

Average CPU Load %

Typical Files LUN #1 Typical Files LUN #1-4 Typical Files LUN #1-4 Small Files LUN #1 Small Files Initial Run LUN #1-4 Small Files LUN #1-4

Backup to Tape

Table 9. Backup to SDLT320
SDLTDrive Transfer Rate (MB/s)

28,84 N/A N/A N/A 12,56

6 N/A N/A N/A 20
SDLTDrives Transfer Rate (MB/s)

N/A N/A 62,87 N/A N/A

N/A N/A 16 N/A N/A
Typical Files LUN #1 Typical Files LUN #1-4 Typical Files LUN #1-4 Small Files LUN #1 Small Files LUN #1-4
Table 10. Backup to Ultrium 460
Ultrium 460 Transfer Rate (MB/s)

39,39 N/A N/A N/A N/A

7 N/A N/A N/A N/A
Table 11. Backup to Ultrium 960
Ultrium 960 Block Size 64 KB Transfer Rate (MB/s)

N/A N/A 91,15 N/A N/A

N/A N/A 23 N/A N/A
Ultrium 960 Block Size 256 KB Transfer Rate (MB/s)
43,01 66,52 91,97 N/A 14,24

11 N/A 21

Advanced Backup to Disk
MSA1000 to File Library (MSA1500)
Table 12. MSA1000 to File Library (MSA1500)
File Library 2 Drives Transfer Rate (MB/s)
Typical Files LUN #1-4 Block Size 256 KB File Depot Size 50 GB Typical Files LUN #1-4 Block Size 256 KB File Depot Size 10 GB Small Files LUN #1-4 Block Size 256 KB File Depot Size 50 GB
File Library (MSA1500) to Tape
Table 13. File Library (MSA1500 RAID 1+0 and 3 Logical Arrays) to SDLT320

Small Files LUN #1-4

Table 14. File Library (MSA1500 RAID 1+0 and 3 Logical Arrays) to Ultrium 460/960
Ultrium 960 Transfer Rate (MB/s)
Table 15. File Library (MSA1500 RAID 5 and 2 Logical Arrays) to Ultrium 960
Typical Files LUN #1-4 Small Files LUN #1-4

45,06 43,37

Restore
File Library (MSA1500) to MSA1000
Table 16. Restore from File Library (MSA1500) to MSA1000
Typical Files LUN #1-4 Small Files Single 8 KB File

RAID 1 was only tested in the 1 logical array configuration, which resulted in poor performance values (write 9,24 MB/s and read 15,72 MB/s). Furthermore, it must be considered that RAID 1 reduces the usable disk space by 50%. RAID 1+0 achieved better results than RAID 1 (write 16,96 MB/s and read 29,26 & 48,30 MB/s) due to striped disk data but the problem of inefficient disk usage remains. RAID 5 was only tested in the configuration with 2 logical arrays to follow the disk array technology with 2 SCSI/SATA ASICs. 6 physical disks were grouped into one logical array. If writing to 1 LUN, the transfer rate showed 21,33 MB/s. If writing simultaneously to 2 LUNs, the transfer rate increased to 34,43 MB/s. If reading from a single LUN, the read transfer rate was 36,57 MB/s. If simultaneously reading from 2 LUNs, the read transfer rate was 56,64 MB/s. The RAID 5 level is chosen for further testing. The results give a good estimate what the MSA1500 staging area (file library) is able to deliver. The maximum read performance of 56,64 MB/s is a good prerequisite for utilizing two SDLT 320 (2*35 MB/s, 2:1 comp.) and Ultrium 460 (60 MB/s, 2:1 comp.). But it shows similar to the MSA1000, that this best-case scenario cannot completely utilize the high-performance Ultrium 960 (157 MB/s, 2:1 comp.).
The backup to the NULL device is a good proof point for the maximum read performance and which tape drive technology would fit to it. The best typical file backup was achieved with the concurrency of 4. The transfer rate of 92,80 MB/s matches almost the MSA1000 read test result of 93,58 MB/s. The results for small files are similar. Please check Table6 and Table8. The initial backup causes Data Protector to save file information into the internal database, which requires more system resources. The initial backup of 22 million small files showed that even such a high number of files have only little impact on the CPU load (27% instead of 23%). Please check Table8.
The SDLT 320 saved the data from a single LUN to tape with 28,84 MB/s, the Ultrium 460 with 39,39 MB/s and the Ultrium 960 with 43,01 MB/s. Both Ultrium (460/960) were slowed down due to the limited disk read performance of 45,70 MB/s. Please check Table6, Table9, Table10 and Table11. The Ultrium 960 was tested with different block sizes (64 and 256 KB). The data transfer rate was comparable (91,15 and 91,97 MB/s) but the average CPU load was much different (23% and 11%). Finally, the increased block size of 256 KB reduced the CPU utilization by 50%. Please check Table11 and the figures below.
Figure 33. Backup to Ultrium 960 - Average CPU Utilization with 64 KB Block Size
Figure 34. Backup to Ultrium 960 - Average CPU Utilization with 256 KB Block Size
These results confirm the block size recommendation of the Ultrium 960 white paper Getting the most performance from your HP StorageWorks Ultrium 960 tape drive white paper (downloadable from http://h18006.www1.hp.com/storage/tapewhitepapers.html). For the typical file backup, the single Ultrium 960 tape drive showed a lower CPU utilization (11%) than double SDLT 320 tape drives (16%). Additionally, the Ultrium 960 showed also a better transfer rate (91,97 MB/s instead of 62,87 MB/s). This and other tests demonstrate that single highperformance tape drives require less CPU resources with the same or even better transfer rate. Please check Table9 and Table11.

The Advanced Backup to Disk performance relies basically on the disk arrays and tape drives source and target. It must be taken into consideration that the MSA1500 provides less performance than the MSA1000 and both less than one Ultrium 960 tape drive. Please note that the MSA1500
and its SATA disks have also advantages. It enables you to improve small file backups and fast single file restores. If typical files were saved from MSA1000 to MSA1500, the maximum transfer rate was 19,47 MB/s. If saved from MSA1500 to Ultrium 960, the transfer rate was 45,06 MB/s. The transfer rate was higher, if the same files were directly saved from the MSA1000 to 2 SDLT 320 (62,87 MB/s) or 1 Ultrium 960 (91,97 MB/s). In this case, tape was faster than disk. Please check Table9, Table11, Table12 and Table15. If small files were saved from MSA1000 to MSA1500, the maximum transfer rate was (12,22 MB/s). The transfer rate was slightly higher, if the same files were directly saved from the MSA1000 to SDLT 320 (12,56 MB/s) or Ultrium 960 (14,24 MB/s). But if saved from MSA1500 to Ultrium 960, the transfer rate increased to 43,37 MB/s. Please check Table9, Table11, Table12 and Table15. These results confirm that Advanced Backup to Disk improves small file backups and tape drive utilization by first staging to disk and then copying to tape. Furthermore the backup of small files was tested with the slower RAID 1+0 and 3 Logical Arrays configuration to demonstrate the impact on the tape drives. If the disk array cannot provide data fast enough, the tape drive suffers from stopping and starting (start/stop mode), which shortens the media life through a large number of repositions (media passes). Therefore, slower tape drives stop less and could be faster than their faster counterparts. In this setup, double SDLT 320 showed the highest transfer rate with 48,55 MB/s. Ultrium 460 showed 42,33 MB/s and Ultrium 960 just 39,25 MB/s. The effect on the average CPU load was opposite. SDLT 320 showed 19%, Ultrium 460 17% and Ultrium 960 just 8%. Please check Table13 and Table14. The file depot size plays an important role for the CPU utilization and the resulting performance. For the typical file backup (to file library), the default Maximum File Depot Size of 50 GB resulted in a transfer rate of 19,47 MB/s with an average CPU utilization of 12%. The reduced Maximum File Depot Size of 10 GB resulted in a transfer rate of 19,24 MB/s but with an average CPU utilization of just 4%. Please check the next two figures and Table12.

Fragmentation of File Library File Systems
Fragmentation of Windows NTFS file systems can decrease the file library performance. For best performance, each file system should belong to one file library writer only. No other writer, process or application should write to it. The following examples show one optimal and one problematic configuration. The optimal configuration with one writer results in very little fragmentation and good read performance as shown in the next three figures.
Figure 37. Optimal File Library Configuration One Writer per File System
Figure 38. Optimal File Library Configuration Little Fragmentation
Figure 39. Optimal File Library Configuration Good Read Performance
The problematic configuration with multiple writers assigned to one single directory results in large fragmentation and poor read performance.
Figure 40. Problematic File Library Configuration Multiple Writers per File System
Figure 41. Problematic File Library Configuration Large Fragmentation
Figure 42. Problematic File Library Configuration Poor Read Performance

Summary and Conclusions

This white paper briefly describes performance-related information for HP OpenView Storage Data Protector 5.5 and the Advanced Backup to Disk feature: The test environment is able to provide a good performance with a low usage of CPU and memory resources. Only a high number of small files cause I/O problems. Disk staging acts as a buffer allowing media drives to operate at maximum speeds and provide the option to do automatic data replication during off-peak hours. This technique is highly recommended when backing up numerous small files to prevent poor transfer rates to tape drive. Disk technologies (SATA, SCSI) and RAID levels could have major performance differences. RAID 0 should not be considered due to missing fault tolerance features. The backup device in this case disk must be very reliable. In general, backups should be available for disasters and selective restores. If a disk fails during the backup or between the backup and the restore (disks always spin, tapes not), backup data would be lost. Backup and restores of typical files could be faster with tape technology than with disk technology (file library). Single file restores are executed with an excellent performance by disk technologies. This is very helpful for selective file restores (particularly multiple times) where time is an important issue. No tape must be loaded and positioned which is a major advantage against tape technologies. The performance can be determined by the performance tools described in chapters HPReadData and Library and Tape Tools. It is recommended to check the disk performance before Data Protector is configured. In most cases, an optimized disk array configuration has more impact than default parameter changes of Data Protector.

For more information

HP OpenView Storage Data Protector
http://www.hp.com/go/dataprotector
HP OpenView Storage Data Protector 5.5 Guides
http://ovweb.external.hp.com/lpe/doc_serv
HP OpenView Storage Data Protector 5.5 Support Matrices HP OpenView Storage Data Protector 5.5 White Papers
http://www.openview.hp.com/products/datapro/spec_0001.html http://www.openview.hp.com/sso/searchdocs?prod=DATAPRO&ct=TWP
HP OpenView Storage Data Protector 5.5 Disk-Assisted Backup White Paper
http://www.openview.hp.com/sso/searchdocs?prod=DATAPRO&ct=TWP http://h18006.www1.hp.com/storage/tapewhitepapers.html
Getting the most performance from your HP StorageWorks Ultrium 960 tape drive white paper HP Performance Assessment Tools http://www.hp.com/support/pat Library and Tape Tools http://www.hp.com/support/tapetools
2003 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Itanium is a trademark or registered trademark of Intel Corporation in the U.S. and other countries and is used under license. XXXX-XXXXEN, 07/2003