HP ProLiant DL585 G5 server technology

technology brief

Abstract..... 2 Introduction..... 2 Operating system support.... 2 DL585 G5 architecture.... 3 Quad-core AMD Opteron processors.... 4 nVidia NForce Professional 2200 and 2050 chipsets... 4 Memory technologies.... 5 I/O technologies.... 6 Storage technologies.... 7 Serial Attached SCSI (SAS)... 7 Smart Array SAS P400 Controller... 7 Networking technologies.... 7 TCP Offload Engine..... 7 Receive Side Scaling (RSS)... 8 Accelerated iSCSI.... 8 Server management technologies... 8 Power management technology.... 9 Conclusion..... 9 For more information.... 10 Call to action.... 10

Abstract

The fifth generation (G5) HP ProLiant DL585 server offers the performance and efficiency of quad-core AMD Opteron processors, enhanced by improvements to all major subsystems in the server. This paper explains the technologies implemented in the HP ProLiant DL585 G5 server that make it an excellent choice for enterprise customers needing increased performance and investment protection with a 64-bit migration path. This paper is intended for IT professionals familiar with system administration and HP ProLiant industry-standard servers. It discusses only key technologies of the ProLiant 500-series servers.

Introduction

For the HP ProLiant DL585 G5 server, HP improved upon the design of previous DL585 generations by integrating quad-core processors and enhancing key subsystems. The DL585 G5 provides excellent system performance by using powerful processors, on-chip memory controllers, a large memory footprint, and fully redesigned I/O and disk subsystems. This server is particularly well suited for the following computing environments: Datacenter virtualization and consolidation projects High-performance technical computing (HPTC) Electronic design automation (EDA)/semiconductor design Financial applications Large database applications ERP/CRM Petrochemical applications Life science and material science applications Video rendering applications

Operating system support

The DL585 G5 supports the following operating systems and virtualization software: Microsoft Windows Server 2003 R2 Standard (32- and 64-bit), Enterprise (32- and 64-bit), Compute Cluster Server, and Datacenter editions Microsoft Windows Server 2008 (when available) Red Hat Enterprise Linux 4 and 5 SuSE Linux Enterprise Server 9 and 10 VMWare ESX virtualization software Microsoft Virtual Server Xen virtualization software 1 For a complete and up-to-date listing of supported OSs and versions, visit the ProLiant OS Support Matrix at http://www.hp.com/go/supportos.
Xen virtualization software is an open source project that allows multiple virtual machines to share the physical hardware of a single host machine. Some virtualization products run virtual machines as guests on top of a host operating system (which requires a large amount of overhead). By contrast, Xen operates as a thin layer directly on top of the underlying hardware. Xen coordinates the interaction of the virtual machines and the physical hardware. HP is a key participant in the development of Xen. For more information about the Xen project, see the XenSource website at www.xensource.com.

DL585 G5 architecture

The DL585 G5 is a powerful, 4U enterprise server incorporating technologies that extend the capabilities of industry-standard x86 computing. This server provides high performance and the capability of running both 32-bit and 64-bit applications simultaneously with no performance penalty when using an operating system that supports 64-bit extensions. This is possible because the AMD64 instruction set is a super-set containing the x86 instruction set architecture. The DL585 G5 architecture includes the following essential features: Support for up to four quad-core AMD Opteron 8300-series processors nVidia nForce Professional 2200 and 2050 chipsets Support for Dual Dynamic Power Management Two 100MHz PCI-X slots Four PCI Express (PCIe) x4 slots Three PCIe x8 slots Two embedded 10/100/1000 Ethernet ports with iSCSI and Receive Side Scaling (RSS) HP Integrated Lights-Out (iLO) 2 management

Figure 1. Block diagram of DL585 G5 architecture
8 GB/s coherent HyperTransport

8 GB/s HyperTransport

8 GB/s HyperTransport 64bits @ 100MHz x8
64bits @ 100MHz 1 Gb Gbit Ethernet BC5706c Gbit Ethernet BC5706c

AMD 8132 HT / PCI-x

nVidia nForce Professional 2200 MCP
x8 nVidia nForce Professional 2050 MCP x4 x4 x4

FLASH SIO

Legacy 32bits @ 33MHz Video iLo

USB EIDE SM Bus

The AMD-8132 HyperTransport PCI-X Tunnel provides two PCI-X bridges (see Figure 1). One 16-bit, 1-GHz HyperTransport interface connects to the CPUs. The other 16-bit, 1-GHz HyperTransport interface connects to the downstream nVidia nForce Professional 2050 PCIe Bridge. The AMD-8132 supports two PCI-X bridges operating at 100 MHz. One PCI-X bridge supports two 100-MHz expansion slots (one full length and the other half length). The other PCI-X bridge supports two NC371i gigabit Ethernet interfaces.
Quad-core AMD Opteron processors
The AMD 8300-series quad-core processor includes four separate microprocessor cores on one physical die (Figure 2). AMD quad-core technology delivers high performance and reduced latency for multi-threaded and multi-tasking environments. Quad-core processors provide two primary benefits: Enhanced performance per wattsignificant performance increase while operating within the same power and thermal envelopes as dual-core processors Increased virtualizationon-die memory controller providing optimum performance in virtual machine (VM) environments Increasing the number of processors available to the operating system enables more efficient execution of multiple threads. In addition, a quad-core processor uses less power than two (equivalent) dual-core processors and therefore reduces power usage and heat production. The DL585 G5 server supports the Dual Dynamic Power Management (DDPM) capabilities of the AMD Opteron 8300 processor. DDPM architecture separates the core and memory controller voltage planes. The HP ProLiant DL585 G5 features a split-plane motherboard that separates the voltage feeds to the CPU core and the DDR2 memory controller. This split-plane design allows granular power management of the Opteron processor for peak operating efficiency and increased performance.
Figure 2. AMD Opteron 8300 series quad-core architecture

512 KB L2 Cache

2 MB L3 Cache System Request I/F Crossbar Switch
DDR2 Memory Cntlr Hypertransport I/F

72-bit

Link 1
As an alternative to traditional front-side bus technology, AMD Opteron processors make use of Direct Connect Architecture and HyperTransport technology. Direct Connect architecture is AMDs designation for the coherent HyperTransport connection between processors. It eliminates the bottlenecks inherent in front-side bus technology by integrating the memory controller into the processor and directly connecting CPUs to the I/O subsystem and other processors. HyperTransport is a parallel, point-to-point interconnect that replaces parallel front-side bus technology. A 64-bit architecture has a much larger amount of directly addressable (flat) memory space than a 32-bit processor. The AMD64 instruction set allows the OS to access a flat memory address space greater than 4 GB without incurring the overhead of Physical Address Extensions (PAE). This can result in performance advantages, particularly the ability to use large amounts of memory such as with intensive floating-point calculations used in scientific and engineering modeling programs.

nVidia NForce Professional 2200 and 2050 chipsets
The DL585 G5 makes use of both the nVidia nForce Professional 2200 and 2050 chipsets to offer a large number of expansion slots for enhanced flexibility. The 2050 chipset connects to the AMD-8132 HyperTransport tunnel with a 16-bit, 1-GHz HyperTransport interface and offers 20 lanes of PCIe support divided into three x4 slots and one x8 slot. The 2200 chipset connects directly to the

Link 2

Link 3
processors with an additional 16-bit, 1-GHz HyperTransport connection. In addition to supporting system devices and USB ports, the 2200 offers an additional PCIe x4 slot and two PCIe x8 slots.

Memory technologies

Typical multiprocessor PC server architecture connects IA-32 processors to memory DIMMs by means of a north bridge chip, which provides the memory controller function. However, in AMD Opteron architecture, the memory controller is integrated into the processor chip for enhanced performance. The on-die memory controller reduces memory latency by eliminating the bus contention between memory and I/O cycles on a typical north bridge. In addition, with each processor containing its own memory controller, the aggregate bandwidth for system-accessible memory is scalable in multiprocessor systems such as the DL585 G5. The Opteron processor supports dual-channel memory. 2 Two 64-bit-wide memory channels operate in parallel to provide a 128-bit interface. Since the DL585 G5 supports dual-width memory channels, DIMMs must be installed in pairs. By default, memory in the DL585 G5 operates in a linear configuration, which provides optimum performance for Microsoft Windows operating system and for many applications, such as Microsoft SQL Server. Linear memory is allocated and de-allocated at the thread level, and access is defined on all nodes sequentially. Sequential addresses are assigned to all memory locations starting on node 0, then to all locations on node 1, and so on until memory locations on all nodes have been assigned. For those applications that cannot take advantage of linear memory configuration, DL585 G5 performance may be improved by activating node interleaving. System administrators can activate node interleaving using the HP ROM-Based Setup Utility (RBSU) provided as part of the HP ProLiant Essentials Foundation Pack. Node-interleaving breaks memory into 4-KB addressable entities. Addressing starts with address 0 on node 0 and assigns sequential addresses through address 4095 to node 0, addresses 4096 through 8191 to node 1, addresses 8192 through 12287 to node 2, and addresses 12888 through 16383 to node 3. Address 16384 is assigned to node 0, and the process continues until all memory has been assigned in this fashion. An application that uses a common allocation thread will benefit from node interleaving. The DL585 G5 supports PC2-5300 DDR2 memory at 533 MHz and 667 MHz. When more than four DIMMS are installed on a single memory controller (that is, per processor), the memory bus for that processor will be clocked down to 533 MHz. Table 1 lists available memory configurations for the DL585 G5.

Table 1. Memory configuration options in the DL585 G5 with PC2-5300 DIMMs Max memory capacity 128 GB (256 GB with 8-GB DIMMs, when available) 64 GB DIMM size 512 MB,1 GB, 2 GB, 4 GB 512 MB, 1 GB, 2 GB, 4 GB Maximum DIMMs per processor Memory speed 533 MHz 667 MHz
For more information on memory technology, refer to the HP technology brief Memory Technology Evolution: An Overview of System Memory Technologies available at the following URL http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00256987/c00256987.pdf

I/O technologies

The DL585 G5 supports two types of I/O expansion slots: PCI-X and PCI Express. The server includes two 64-bit, 100-MHz, PCI-X slots (one full length, one half length) that provide a 64-bit, 100-MHz interface for expansion devices. PCI Express (PCIe) is an expansion bus technology that builds upon earlier PCI technology, but it uses a serial interface rather than the traditional parallel PCI bus. A PCIe interface 3 can support one or more lanes, as determined by the PCIe slot. Multiple-lane connections are described as x4 for a 4-lane connection, x8 for an 8-lane connection, and so forth. Each lane is a separate serial connection of four wires and has a bandwidth of 250 MB/s in each direction (for a total bandwidth of 500 MB/s per single PCIe lane link). PCIe links automatically negotiate the highest number of lanes that the slot and the card support. Therefore, a PCIe expansion card can be installed into any slot in which it will physically fit and it will work correctly. The DL585 G5 has seven PCIe slots: three full-length x8 slots, three full-length x4 slots, and one halflength x4 slot. Since PCIe interface handling is split 4/3 between the two processor pairs and their associated chipsets, optimal performance can be achieved by balancing the load of multiple PCIe expansion cards across the HyperTransport links. Table 2 identifies the preferred order of installation.
Table 2. Preferred order for populating PCIe expansion slots in the ProLiant DL585 G5 server Installation sequence First Second Third Fourth Fifth Sixth Seventh Slot 6 Speed X8 X8 X8 X4 X4 X4 X4
For more information on PCIe technology, refer the white paper HP Local I/O Technology for ProLiant and BladeSystem Servers available at the following URL: http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00231623/c00231623.pdf

Storage technologies

The DL585 G5 has an updated disk subsystem and includes the Smart Array SAS P400 Controller. Serial Attached SCSI (SAS) The peak data transfer rate for the current generation of SAS drives 4 is 3 gigabits per second in full duplex mode. In a serial system, the devices do not share a common bus, and each device can take full advantage of the bandwidth available to it. The SAS backplane connector is compatible with SATA drives (although it is not possible to use a SAS drive on a SATA controller), and the chipset on a SAS controller can communicate with SATA drives. This enables administrators to store data that needs to be available at a high performance standard on SAS disk drives, and to use the same enclosure to store data on slower but higher capacity SATA drives. Smart Array SAS P400 Controller The DL585 G5 includes a Smart Array P400 Controller which supports SAS storage. Some configurations of the DL585 G5 include a battery-backed write cache (BBWC) on the P400 controller. For models that do not include the BBWC, it is available as an option. The battery will last two days without receiving any power from the computer. The BBWC buffers disk writes so that disk I/O can be handled efficiently. If there is an unexpected system shutdown, the battery maintains the information in the buffer so that data will not be lost. In the event of a complete system failure, the controller and disks can be moved to a different server, where the controller will flush out the cache to the disks once power has been restored. If a controller fails, the cache module and disks can be removed from the failed controller and installed on a working controller, where the cache will be flushed out to the disks. The Smart Array P400 supports RAID levels 0, 1, 1+0 and 5. RAID 6 (double parity) is available and does require the BBWC. Mirror splitting is available for RAID 1 arrays. This functionality allows the user to split a RAID 1 mirror into two separate RAID 0 arrays (breaking the mirror). Mirror recombining is the opposite combining two RAID 0 arrays into a RAID 1 mirror. BBWC is not required for mirror recombining. The BBWC is required for capacity expansion functionality, which allows the user to add a physical disk to an existing array. The controller then recalculates parity and balances the data across the disks. During the expansion, data and logical structures on the array are preserved. The P400 controller supports a recovery ROM that allows for failing back to the previous valid ROM in the event of a failed attempt to flash the controllers ROM. Online Drive Flashing is also supported. With Online Drive Flashing, disk drive firmware updates can be pre-loaded onto the controller, and then at the next reboot, the controller will flash the firmware on the hard drives.

Networking technologies

The DL585 G5 includes two BC5706i multifunction gigabit Ethernet (GbE) network adapters. Technologies supported by the BC5706i include TCP Offload Engine, Receive Side Scaling (RSS), and accelerated iSCSI. TCP Offload Engine Network bandwidth has improved steadily since the early days of TCP/IP networking. Along with this improvement in speed has come increased demand for CPU cycles to manage the network protocol
For more information about SAS technology, refer to the HP technology brief titled Serial Attached SCSI technology available at http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00302340/c00302340.pdf.
stack. A busy, full-duplex gigabit Ethernet connection can consume the available computing power of a 1-GHz Pentium 4 processor. Unfortunately, this means that even a modern, high-powered processor will show degraded performance in processing application instructions while data is being transferred onto or off of the network. Computers most susceptible to this problem are application, web, and file servers that have a high number of concurrent connections. To reduce this burden on the processor, the embedded network controller in the DL585 G5 is designed with TCP Offload Engine (TOE) capabilities. TOE NICs (network interface cards) are designed with on-board logic to process common and repetitive tasks of TCP network traffic. Because the processor does not have to devote cycles to processing these tasks, it can be used more efficiently to significantly increase application performance on servers attached to gigabit Ethernet networks. TOE is supported on Microsoft Windows Server 2003 with the Scalable Networking Pack installed. Receive Side Scaling (RSS) The Network Driver Interface Specifications (NDIS) define a common Application Programming Interface for network interface cards on Microsoft operating systems. Early versions of NDIS did not differentiate between computers with single or multiple processors. The result was that one processor was forced to handle the entire network processing load. NDIS v6.0 includes support for multiple processors. With NDIS v6.0 support, the DL585 G5 can dynamically balance the processing of network packets received across multiple processors. The Scalable Networking Pack for Windows Server 2003 is required for RSS support. Accelerated iSCSI iSCSI (internet SCSI) is a standard that implements the SCSI protocol for interacting with storage devices over a TCP/IP network. While iSCSI can be implemented over any TCP/IP network, the most common implementation is over gigabit Ethernet. iSCSI serves the same purpose as Fibre Channel in building Storage Area Networks (SANs), but without the cost, complexity, and compatibility issues associated with Fibre Channel SANs. A more efficient approach is to offload the management of the protocol to the network adapter. Adapters that implement the iSCSI protocol are known as iSCSI HBAs. The embedded NIC of the DL585 G5 is an iSCSI HBA and appears to the operating system as a SCSI HBA.

Server management technologies
The HP ProLiant DL585 G5 server includes the HP Integrated Lights-Out 2 (iLO 2) management technology 5. iLO 2 hardware and firmware provide remote server management capabilities over Ethernet. The iLO 2 management processor obtains its power from the auxiliary power plane of the server, so it is always available when the server is plugged into a power sourceeven when the OS not operating. The DL585 G5 offers the ability to connect to iLO 2 through a dedicated Ethernet port. This port can connect to a dedicated management network that is out of the data path for the server and that can be highly secured. Alternatively, in the DL585 G5, iLO 2 can assign an additional MAC and IP address to one of the standard embedded network interfaces and share that port with normal traffic. This shared arrangement eliminates the need for an extra network connection for iLO 2.
For more information about iLO 2, refer to the HP technology brief titled Integrated Lights-Out technology: enhancing the manageability of ProLiant servers available at http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00257345/c00257345.pdf.
Power management technology
The HP DL585 G5 includes HP Power Regulator, an innovative OS-independent power management tool. HP Power Regulator is a ROM-based utility that can be used to set the server to one of four power modes: Static high power Server runs continuously in the highest performance state. Static low power Server runs continuously in the lowest power state. OS Control Server uses AMD Opteron PowerNow! Technology that allows the operating system or drivers to control processor power states. Dynamic power savings (new with the DL585 G5) server processor power is automatically adjusted according to application activity. HP Power Regulator improves the energy efficiency of the DL585 G5 by allowing the Opteron processors to run at full power when they need it and enabling power savings without performance degradation when application activity is reduced. HP Power Regulator modes can be selected through the ROM-based Setup Utility during POST, or through the iLO 2 remote management interface anytime during runtime. For more information about HP Power Regulator, see the paper titled Power Regulator for ProLiant servers, available at http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00593374/c00593374.pdf.

Conclusion

The HP ProLiant DL585 G5 is a 4U rack-optimized, 4-processor server created for large data center deployments requiring enterprise-class performance, uptime, and scalability, plus ease of management and expansion. It offers customers running both 32- and 64-bit applications increased performance and memory speed. Quad-core processor support and improved network and I/O subsystems ensure that the DL585 G5 is scalable in high-performance computing environments.

For more information

For additional information, refer to the resources listed below.

Type of information Explanation of how HyperTransport technology works and the advantages it offers for system design and performance Up-to-date information on operating systems and versions supported by the HP ProLiant DL585 G5 server Latest drivers available for the HP ProLiant DL585 G5 server Details about HP Systems Insight Manager Details about iLO management functionality Details about HP ProLiant Essentials Information about SAS technology
Source AMD white paper HyperTransport Technology: Simplifying System Design, October 2002 www.hypertransport.org/docs/26635A_HT_System_Design.p df http://www.hp.com/go/supportos
www.hp.com/support/files www.hp.com/go/hpsim http://h20000.www2.hp.com/bc/docs/support/SupportMan ual/c00257345/c00257345.pdf www.hp.com/servers/proliantessentials http://h20000.www2.hp.com/bc/docs/support/SupportMan ual/c00302340/c00302340.pdf

Call to action

Send any comments about this paper to TechCom@HP.com.
2006, 2008 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. AMD and Opteron are trademarks and AMD-8000 is a mark of Advanced Micro Devices, Inc. HyperTransport is a licensed trademark of the HyperTransport Technology Consortium. Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. Linux is a U.S. registered trademark of Linus Torvalds. TC080203TB, February 2008

HP Microsoft Exchange Server 2003 Example Configuration
1000 User, HP ProLiant DL585 G2, HP StorageWorks MSA1510i
Item Users Server Storage Availability Mailbox size Performance Storage Groups / DB size Assumptions Detail 1000 Mailbox/Front End: DL585 G2 with 4GB MSA1510i with MSA30 Microsoft Clustering: No 165 MB.7 IOPs/user - medium 2 SGs, 6 MDBs total / 45GB databases System, Logs and Databases on same disks (RAID10)
Quan. Part Number 10 413929-001 375861-B21 HA114A1 5A1 HA104A3 AD168A HA114A1 AD537A 254786-B21 AD539A 302969-B21 286778-B22 Description HP DL585 G2.8 2P PC5300 US Svr HP 72GB 10K SAS Hot Plug Hard Drive 500 Series Hardware Installation HP 3y 4h 24x7 HW Support (included) HP FC2243 4GB PCI-X 2.0 DCHBA HP Installation and Startup Services (included) MSA1510i controller shelf HP 256MB Battery-Backed Cache HP MSA1510i Controller Module HP StorageWorks MSA30 SB Storage HP 72GB 15k U320 Pluggable Hard Drive

ProLiant DL585 G2

MSA1510i
HP Microsoft Exchange Server 2003 Example Configuration Assumptions
Example configuration based on best practices, extrapolation based on current test knowledge, and performance assumptions through discussions with Microsoft and HP resources This is an example; there are many additional server/storage options which would also meet the requirements User profiles
.7 IOPS/u; "medium" user profile (CPU burn rates) 1.6 x multiplier for deleted items, maintenance and recovery space ~90% fill factor on database disks Assumes 100% concurrency Minimum 165MB mailboxes

Storage choices

Unless otherwise noted, RAID10 is used for databases, and logs and databases on separate spindles (above 100 users)
No consideration in this example for backup solutions, spares (unless otherwise noted), recovery storage groups, VSS plexes, etc.