Performance Best Practices for VMware vSphere 4.0
VMware ESX 4.0 and ESXi 4.0 vCenter Server 4.0

EN-000005-03

You can find the most up-to-date technical documentation on the VMware Web site at: http://www.vmware.com/support/ The VMware Web site also provides the latest product updates. If you have comments about this documentation, submit your feedback to: docfeedback@vmware.com
2007-2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware, the VMware boxes logo and design, Virtual SMP, and VMotion are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
VMware, Inc. 3401 Hillview Ave. Palo Alto, CA 94304 www.vmware.com

VMware, Inc.

Contents

About This Book

1 Hardware for Use with VMware vSphere 11
Validate Your Hardware 11 Hardware CPU Considerations 11 Hardware-Assisted Virtualization 11 Hardware-Assisted CPU Virtualization (Intel VT-x and AMD AMD-V) 11 Hardware-Assisted MMU Virtualization (Intel EPT and AMD RVI) 11 Hardware Storage Considerations 13 Hardware Networking Considerations 14 Hardware BIOS Settings 15
2 ESX and Virtual Machines 17
ESX General Considerations 17 ESX CPU Considerations 19 Hyper-Threading 20 Non-Uniform Memory Access (NUMA) 21 Configuring ESX for Hardware-Assisted Virtualization 21 ESX Memory Considerations 23 Memory Overhead 23 Memory Sizing 23 Memory Overcommit Techniques 23 Memory Swapping 24 Large Memory Pages for Hypervisor and Guest Operating System 25 Hardware-Assisted MMU Virtualization 25 ESX Storage Considerations 26 ESX Networking Considerations 28
3 Guest Operating Systems 29
Guest Operating System General Considerations 29 Running Paravirtualized Operating Systems 29 Measuring Performance in Virtual Machines 30 Guest Operating System CPU Considerations 31 Guest Operating System Storage Considerations 32 Guest Operating System Networking Considerations 33
4 Virtual Infrastructure Management 35
General Resource Management Best Practices 35 VMware vCenter Best Practices 36 VMware vCenter Database Considerations 36 VMware VMotion and Storage VMotion 37 VMware Distributed Resource Scheduler (DRS) Best Practices 38 VMware Distributed Power Management (DPM) Best Practices 40 VMware Fault Tolerance Best Practices 41

Glossary Index 51

Tables
Table 1. Conventions Used in This Manual 8 Table 2-1. Symptoms of Insufficient Resources for the ESX Service Console
This book, Performance Best Practices for VMware vSphere 4.0, provides a list of performance tips that cover the most performance-critical areas of VMware vSphere 4.0. It is not intended as a comprehensive guide for planning and configuring your deployments. Chapter 1, Hardware for Use with VMware vSphere, on page 11, provides guidance on selecting hardware for use with vSphere. Chapter 2, ESX and Virtual Machines, on page 17, provides guidance regarding VMware ESX software and the virtual machines that run in it. Chapter 3, Guest Operating Systems, on page 29, provides guidance regarding the guest operating systems running in virtual machines. Chapter 4, Virtual Infrastructure Management, on page 35, provides guidance regarding resource management best practices. NOTE For planning purposes we recommend reading this entire book before beginning a deployment. Material in the Virtual Infrastructure Management chapter, for example, may influence your hardware choices. NOTE Most of the recommendations in this book apply to both ESX and ESXi. The few exceptions are noted in the text.

For technical papers providing more detail on topics discussed in this book refer to http://www.vmware.com/vmtn/resources.
Issue Idle loop CPU utilization Network throughput between virtual machines Queue depths on QLogic cards Guest storage drivers Disk outstanding commands parameter VMotion CPU Compatibility Requirements for Intel Processors
Publication KB article 1730 KB article 2032 KB article 1428 KB article 1267 KB article 9645697 KB article 1268 KB article 1991
VMotion CPU Compatibility KB article 1992 Requirements for AMD Processors Migrations with VMotion Prevented Due to CPU MismatchHow to Override Masks Detailed explanation of VMotion considerations Timing in virtual machines SAN best practices Memory allocation Swap space Supported maximums KB article 1993
Whitepaper: VMware VMotion and CPU Compatibility Whitepaper: Timekeeping in VMware Virtual Machines Fibre Channel SAN Configuration Guide iSCSI SAN Configuration Guide vSphere Resource Management Guide vSphere Resource Management Guide VMware vSphere 4 Release Notes Configuration Maximums for VMware vSphere 4

Hardware requirements

ESX and vCenter Server Installation Guide
VMware vCenter Update Manager Whitepaper: VMware vCenter Update Manager Performance and Best Practices
Hardware for Use with VMware vSphere
This chapter provides guidance on selecting and configuring hardware for use with VMware vSphere.

Validate Your Hardware

Test system memory for 72 hours, checking for hardware errors. Verify that all hardware in the system is on the hardware compatibility list for the VMware ESX version you will be running. Make sure that your hardware meets the minimum configuration supported by the VMware ESX version you will be running.
Hardware CPU Considerations
When purchasing hardware, it is a good idea to consider CPU compatibility for VMware VMotion and VMware Fault Tolerance. See VMware vCenter Best Practices on page 36.
Hardware-Assisted Virtualization
Many recent processors from both Intel and AMD include hardware features to assist virtualization. These features were released in two generations: the first generation introduced CPU virtualization; the second generation included CPU virtualization and added memory management unit (MMU) virtualization. For the best performance, make sure your system uses processors with second-generation hardware-assist features.
Hardware-Assisted CPU Virtualization (Intel VT-x and AMD AMD-V)
The first generation of hardware virtualization assistance, VT-x from Intel and AMD-V from AMD, became available in 2006. These technologies automatically trap sensitive calls, eliminating the overhead required to do so in software. This allows the use of a hardware virtualization (HV) virtual machine monitor (VMM) as opposed to a binary translation (BT) VMM.
Hardware-Assisted MMU Virtualization (Intel EPT and AMD RVI)

Some recent processors also include a new feature that addresses the overheads due to memory management unit (MMU) virtualization by providing hardware support to virtualize the MMU. ESX 4.0 supports this feature in both AMD processors, where it is called rapid virtualization indexing (RVI) or nested page tables (NPT), and in Intel processors, where it is called extended page tables (EPT). Without hardware-assisted MMU virtualization, the guest operating system maintains guest virtual memory to guest physical memory address mappings in guest page tables, while ESX maintains shadow page tables that directly map guest virtual memory to host physical memory addresses. These shadow page tables are maintained for use by the processor and are kept consistent with the guest page tables. This allows ordinary memory references to execute without additional overhead, since the hardware translation lookaside buffer (TLB) will cache direct guest virtual memory to host physical memory address translations read from the shadow page tables. However, extra work is required to maintain the shadow page tables.

VMware, Inc. 11

Hardware-assisted MMU virtualization allows an additional level of page tables that map guest physical memory to host physical memory addresses, eliminating the need for ESX to intervene to virtualize the MMU in software. For information about configuring the way ESX uses hardware virtualization, see Configuring ESX for Hardware-Assisted Virtualization on page 21.
Chapter 1 Hardware for Use with VMware vSphere
Hardware Storage Considerations
Back-end storage configuration can greatly affect performance. Refer to the following sources for more information: For SAN best practices: Using ESX Server with SAN: Concepts in the SAN Configuration Guide. For NFS best practices: Advanced Networking in the Server Configuration Guide. For iSCSI best practices: Configuring Storage in the Server Configuration Guide. For a comparison of Fibre Channel, iSCSI, and NFS: Comparison of Storage Protocol Performance. Storage performance issues are most often the result of configuration issues with underlying storage devices and are not specific to ESX. Storage performance is a vast topic that depends on workload, hardware, vendor, RAID level, cache size, stripe size, and so on. Consult the appropriate documentation from VMware as well as the storage vendor. Many workloads are very sensitive to the latency of I/O operations. It is therefore important to have storage devices configured correctly. The remainder of this section lists practices and configurations recommended by VMware for optimal storage performance. For iSCSI and NFS, make sure that your network topology does not contain Ethernet bottlenecks, where multiple links are routed through fewer links, potentially resulting in oversubscription and dropped network packets. Any time a number of links transmitting near capacity are switched to a smaller number of links, such oversubscription is a possibility. Recovering from dropped network packets results in large performance degradation. In addition to time spent determining that data was dropped, the retransmission uses network bandwidth that could otherwise be used for new transactions. Applications or systems that write large amounts of data to storage, such as data acquisition or transaction logging systems, should not share Ethernet links to a storage device. These types of applications perform best with multiple connections to storage devices. Performance design for a storage network must take into account the physical constraints of the network, not logical allocations. Using VLANs or VPNs does not provide a suitable solution to the problem of link oversubscription in shared configurations. VLANs and other virtual partitioning of a network provide a way of logically configuring a network, but don't change the physical capabilities of links and trunks between switches. For NFS and iSCSI, if the network switch deployed for the data path supports VLAN, it is beneficial to create a VLAN just for the ESX host's vmknic and the NFS/iSCSI server. This minimizes network interference from other packet sources. If you have heavy disk I/O loads, you may need to assign separate storage processors to separate systems to handle the amount of traffic bound for storage. To optimize storage array performance, spread I/O loads over the available paths to the storage (across multiple host bus adapters (HBAs) and storage processors (SPs)). Configure maximum queue depth for HBA cards. For additional information see KB article 1267, listed in Related Publications on page 8. Be aware that with software-initiated iSCSI and NAS the network protocol processing takes place on the host system, and thus these can require more CPU resources than other storage options. In order to use VMware Storage VMotion your storage infrastructure must provide sufficient available storage bandwidth. For the best Storage VMotion performance you should make sure that the available bandwidth will be well above the minimum required. We therefore recommend you consider the information in VMware VMotion and Storage VMotion on page 37 when planning a deployment.

Hardware BIOS Settings

The default hardware BIOS settings on servers might not always be the best choice for optimal performance. This section lists some of the BIOS settings you might want to check. NOTE ESX 4.0 supports Enhanced Intel SpeedStep and Enhanced AMD PowerNow! CPU power management technologies that can save power when a host is not fully utilized. However because these and other power-saving technologies can reduce performance in some situations, you should consider disabling them when performance considerations outweigh power considerations. NOTE Because of the large number of different server models and configurations, any list of BIOS options is always likely to be incomplete. Make sure you are running the latest version of the BIOS available for your system. NOTE After updating the BIOS, you may need to make sure your BIOS settings are still as you wish. Make sure the BIOS is set to enable all populated sockets, and enable all cores in each socket. Enable Turbo Mode if your processors support it. Make sure hyper-threading is enabled in the BIOS. Some NUMA-capable systems provide an option to disable NUMA by enabling node interleaving. In most cases you will get the best performance by disabling node interleaving. Make sure any hardware-assisted virtualization features (VT-x, AMD-V, EPT, RVI) are enabled in the BIOS. NOTE After these changes are made some systems may need a complete power down before the changes take effect. See http://communities.vmware.com/docs/DOC-8978 for details. Disable C1E halt state in the BIOS. (See note above regarding performance considerations versus power considerations.) Disable any other power-saving mode in the BIOS. (See note above regarding performance considerations versus power considerations.) Disable any unneeded devices from the BIOS, such as serial and USB ports.

ESX and Virtual Machines

This chapter provides guidance regarding ESX software itself and the virtual machines that run in it.
ESX General Considerations
This subsection provides guidance regarding a number of general performance considerations in ESX. Plan the deployment. Allocate enough resources, especially (in the case of ESX, but not ESXi) for the service console. Table 2-1 lists symptoms of insufficient resources for the service console. Table 2-1. Symptoms of Insufficient Resources for the ESX Service Console
Insufficient Resource Type Memory Disk Space Symptoms Poor response from the vSphere Client Inability to write diagnostic messages to log; inability to log into the service console
Allocate only as much virtual hardware as required for each virtual machine. Provisioning a virtual machine with more resources than it requires can, in some cases, reduce the performance of that virtual machine as well as other virtual machines sharing the same host. Disconnect or disable unused or unnecessary physical hardware devices, such as: COM ports LPT ports USB controllers Floppy drives Optical drives (that is, CD or DVD drives) Network interfaces Storage controllers Disabling hardware devices (typically done in BIOS) can free interrupt resources. Additionally, some devices, such as USB controllers, operate on a polling scheme that consumes extra CPU resources. Lastly, some PCI devices reserve blocks of memory, making that memory unavailable to ESX. Unused or unnecessary virtual hardware devices can impact performance and should be disabled. For example, windows guests poll optical drives (that is, CD or DVD drives) quite frequently. When virtual machines are configured to use a physical drive, and multiple guests simultaneously try to access that drive, performance could suffer. This can be reduced by configuring the virtual machines to use ISO images instead of physical drives, and can be avoided entirely by disabling optical drives in virtual machines when the devices are not needed.

For nearly all workloads, custom hyper-threading settings are not necessary. In cases of unusual workloads that interact badly with hyper-threading, however, choosing the None or Internal hyper-threading option might help performance. For example, an application with cache-thrashing problems might slow down an application sharing its physical CPU core. In this case configuring the virtual machine running that problem application with the None hyper-threading option might help isolate that virtual machine from other virtual machines. The trade-off for configuring None or Internal should also be considered. With either of these settings, there can be cases where there is no core to which a descheduled virtual machine can be migrated, even though one or more logical cores are idle. As a result, it is possible that virtual machines with hyper-threading set to None or Internal can experience performance degradation, especially on systems with a limited number of CPU cores.
Non-Uniform Memory Access (NUMA)
IBM (X-Architecture), AMD (Opteron-based), and Intel (Nehalem) non-uniform memory access (NUMA) systems are supported in ESX. On AMD Opteron-based systems, such as the HP ProLiant DL585 Server, BIOS settings for node interleaving determine whether the system behaves like a NUMA system or like a uniform memory accessing (UMA) system. For more information, refer to your servers documentation. If node interleaving is disabled, ESX detects the system as NUMA and applies NUMA optimizations. If node interleaving (also known as interleaved memory) is enabled, ESX does not detect the system as NUMA. The intelligent, adaptive NUMA scheduling and memory placement policies in ESX can manage all virtual machines transparently, so that administrators do not need to deal with the complexity of balancing virtual machines between nodes by hand. However, manual override controls are available, and advanced administrators may prefer to control the memory placement (through the Memory Affinity option) and processor utilization (through the Only Use Processors option). By default, ESX NUMA scheduling and related optimizations are enabled only on systems with a total of at least four CPU cores and with at least two CPU core per NUMA node. On such systems, virtual machines can be separated into the following categories: Virtual machines with a number of vCPUs equal to or less than the number of cores in each NUMA node will be managed by the NUMA scheduler and will have the best performance. Virtual machines with more vCPUs than the number of cores in a NUMA node will not be managed by the NUMA scheduler. They will still run correctly, but they will not benefit from the ESX NUMA optimizations. NOTE More information about using NUMA systems with ESX can be found in the Advanced Attributes and What They Do and Using NUMA Systems with ESX Server sections of the VMware Resource Management Guide, listed in Related Publications on page 8.

ESX Memory Considerations
This subsection provides guidance regarding memory considerations in ESX.

Memory Overhead

Virtualization causes an increase in the amount of physical memory required due to the extra memory needed by ESX for its own code and for data structures. This additional memory requirement can be separated into two components: A system-wide memory space overhead for the service console (typically 272MB, but not present in ESXi) and for the VMkernel (typically about 100MB). An additional memory space overhead for each virtual machine.
The per-virtual-machine memory space overhead includes space reserved for the virtual machine devices (e.g., the SVGA frame buffer and several internal data structures maintained by the VMware software stack). These amounts depend on the number of vCPUs, the configured memory for the guest operating system, and whether the guest operating system is 32-bit or 64-bit. ESX provides optimizations such as memory sharing (see Memory Overcommit Techniques on page 23) to reduce the amount of physical memory used on the underlying server. In some cases these optimizations can save more memory than is taken up by the overhead. The VMware Resource Management Guide, listed in Related Publications on page 8, includes a table with detailed memory space overhead numbers.

Memory Sizing

Carefully select the amount of memory you allocate to your virtual machines. You should allocate enough memory to hold the working set of applications you will run in the virtual machine, thus minimizing swapping, but avoid over-allocating memory. Allocating more memory than needed unnecessarily increases the virtual machine memory overhead, thus using up memory that could be used to support more virtual machines. When possible, configure 32-bit Linux virtual machines with no more than 896MB of memory. 32-bit Linux kernels use different techniques to map memory on systems with more than 896MB. These techniques impose additional overhead on the virtual machine monitor and can result in slightly reduced performance.
Memory Overcommit Techniques
ESX uses three memory management mechanisms page sharing, ballooning, and swapping to dynamically reduce the amount of machine physical memory required for each virtual machine. Page Sharing: ESX uses a proprietary technique to transparently and securely share memory pages between virtual machines, thus eliminating redundant copies of memory pages. Page sharing is used by default regardless of the memory demands on the host system. Ballooning: If the virtual machines memory usage approaches its memory target, ESX uses ballooning to reduce that virtual machines memory demands. Using a VMware-supplied vmmemctl module installed in the guest operating system as part of VMware Tools suite, ESX can cause the guest to relinquish the memory pages it considers least valuable. Ballooning provides performance closely matching that of a native system under similar memory constraints. To use ballooning, the guest operating system must be configured with sufficient swap space. Swapping: If ballooning fails to sufficiently limit a virtual machines memory usage, ESX also uses host-level swapping to forcibly reclaim memory from a virtual machine. Because this will swap out active pages, it can cause virtual machine performance to degrade significantly.

Before performing an alignment, carefully evaluate the performance impact of the unaligned VMFS partition on your particular workload. The degree of improvement from alignment is highly dependent on workloads and array types. You might want to refer to the alignment recommendations from your array vendor for further information. Multiple heavily-used virtual machines concurrently accessing the same VMFS volume, or multiple VMFS volumes backed by the same LUNs, can result in decreased storage performance. Appropriately-configured storage architectures can avoid this issue. For information about storage configuration and performance, see Scalable Storage Performance (available at http://www.vmware.com/resources/techresources/1059). Meta-data-intensive operations can impact virtual machine I/O performance. These operations include: Administrative tasks such as backups, provisioning virtual disks, cloning virtual machines, or manipulating file permissions. Scripts or cron jobs that open and close files. These should be written to minimize open and close operations (open, do all that needs to be done, then close, rather than repeatedly opening and closing). Dynamically growing.vmdk files for thin-provisioned virtual disks. When possible, use thick disks for better performance. You can reduce the effect of these meta-data-intensive operations by scheduling them at times when you dont expect high virtual machine I/O load. More information on this topic can be found in Scalable Storage Performance (available at http://www.vmware.com/resources/techresources/1059). I/O latency statistics can be monitored using esxtop (or resxtop), which reports device latency, time spent in the kernel, and latency seen by the guest. Make sure I/O is not queueing up in the VMkernel by checking the number of queued commands reported by esxtop (or resxtop). Since queued commands are an instantaneous statistic, you will need to monitor the statistic over a period of time to see if you are hitting the queue limit. To determine queued commands, look for the QUED counter in the esxtop (or resxtop) storage resource screen. If queueing occurs, try adjusting queue depths. For further information see KB article 1267, listed in Related Publications on page 8. The driver queue depth can be set for some VMkernel drivers. For example, while the default queue depth of the QLogic driver is 32, specifying a larger queue depth may yield higher performance. You can also adjust the maximum number of outstanding disk requests per virtual machine in the VMkernel through the vSphere Client. Setting this parameter can help equalize disk bandwidth across virtual machines. For additional information see KB article 1268, listed in Related Publications on page 8. By default, Active/Passive storage arrays use Most Recently Used path policy. Do not use Fixed path policy for Active/Passive storage arrays to avoid LUN thrashing. For more information, see the VMware SAN Configuration Guide, listed in Related Publications on page 8. By default, Active/Active storage arrays use Fixed path policy. When using this policy you can maximize the utilization of your bandwidth to the storage array by designating preferred paths to each LUN through different storage controllers. For more information, see the VMware SAN Configuration Guide, listed in Related Publications on page 8. Do not allow your service consoles root file system to become full. (Because it does not include a service console, this point doesnt apply to ESXi.) Disk I/O bandwidth can be unequally allocated to virtual machines by using the vSphere Client (select Edit virtual machine settings, choose the Resources tab, select Disk, then change the Shares field).

Running Paravirtualized Operating Systems
ESX includes support for virtual machine interface (VMI), used for communication between the guest operating system and the hypervisor, thus improving performance and efficiency. Enabling this support will improve the performance of virtual machines running operating systems with VMI by reducing their CPU utilization and memory space overhead (the later being especially true for SMP virtual machines). Even when only some of the virtual machines on an ESX system use VMI, the performance of all virtual machines on that server might benefit due to the hardware resources freed up for ESX to allocate elsewhere.

VMware, Inc. 29

ESX VMI support can be enabled for a virtual machine through the vSphere Client by selecting Edit virtual machine settings, choosing the Options tab, selecting Paravirtualization, then marking the box next to Support VMI Paravirtualization. Enabling VMI for a virtual machine reduces the number of available PCI slots in the guest operating system running in that virtual machine by one. There is no performance benefit to enabling VMI for a virtual machine running a non-VMI operating system. Kernel support for VMI is included in some recent Linux distributions (Ubuntu 7.04 and later and SLES 10 SP2, for example), and can be compiled into other Linux distributions, typically by compiling the kernel with CONFIG_PARAVIRT and CONFIG_VMI. No Microsoft Windows operating systems support VMI. Check the VMware Guest Operating System Installation Guide to see which VMI operating systems are supported in ESX. More information about VMI can be found in Performance of VMware VMI (http://www.vmware.com/resources/techresources/1038) and the Paravirtualization API Version 2.5 (http://www.vmware.com/pdf/vmi_specs.pdf). For best performance, consider the following regarding enabling VMI support: If running 32-bit Linux guest operating systems that include kernel support for VMI on hardware that does not support hardware-assisted MMU virtualization (EPT or RVI), enabling VMI will improve performance. VMI-enabled virtual machine always use Binary Translation (BT) and shadow page tables, even on systems that support hardware-assisted MMU virtualization. Because hardware-assisted MMU virtualization almost always provides more of a performance increase than VMI, we recommend disabling VMI for virtual machines running on hardware that supports hardware-assisted MMU virtualization. (No kernel change is required in the guest, as the VMI kernel can run in a non-VMI enabled virtual machine.)

Measuring Performance in Virtual Machines
Be careful when measuring performance from within virtual machines. Timing numbers measured from within virtual machines can be inaccurate, especially when the processor is overcommitted. NOTE One possible approach to this issue is to use a guest operating system that has good timekeeping behavior when run in a virtual machine, such as a guest that uses the NO_HZ kernel configuration option (sometimes called tickless timer) or the VMI paravirtual timer. More information about this topic can be found in Timekeeping in VMware Virtual Machines (http://www.vmware.com/pdf/vmware_timekeeping.pdf). Measuring performance from with virtual machines can fail to take into account resources used by ESX for tasks it has offloaded from the guest operating system, as well as resources consumed by virtualization overhead. For additional information see KB article 2032, listed in Related Publications on page 8. Measuring resource utilization using tools at the ESX level, such as the vSphere Client, VMware Tools, esxtop, or resxtop can avoid these problems.
Chapter 3 Guest Operating Systems
Guest Operating System CPU Considerations
In SMP guests the guest operating system can migrate processes from one vCPU to another. This migration can incur a small CPU overhead. If the migration is very frequent it might be helpful to pin guest threads or processes to specific vCPUs. (Note that this is another reason not to configure virtual machines with more vCPUs than they need.) Many operating systems keep time by counting timer interrupts. The timer interrupt rates vary between different operating systems and versions. For example: Unpatched 2.4 and earlier Linux kernels request timer interrupts at 100 Hz (100 interrupts per second). Older 2.6 Linux kernels and some 2.4 Linux kernels request interrupts at 1000 Hz. Newer 2.6 Linux kernels request interrupts at 250 Hz. The most recent 2.6 Linux kernels introduce the NO_HZ kernel configuration option (sometimes called tickless timer) that uses a variable timer interrupt rate. Microsoft Windows operating system timer interrupt rates are specific to the version of Microsoft Windows and the Windows HAL that is installed. Windows systems typically use timer interrupt rates of between 66 Hz and 100 Hz. Running applications that make use of the Microsoft Windows multimedia timer functionality can increase the timer interrupt rate. For example, some multimedia applications or Java applications increase the timer interrupt rate to 1000 Hz. The total number of timer interrupts delivered to the virtual machine depends on a number of factors: Virtual machines running SMP HALs/kernels (even if they are running on a UP virtual machine) require more timer interrupts than those running UP HALs/kernels. The more vCPUs a virtual machine has, the more interrupts it requires. Delivering many virtual timer interrupts negatively impacts guest performance and increases host CPU consumption. If you have a choice, use guest operating systems that require fewer timer interrupts. For example: If you have a UP virtual machine use a UP HAL/kernel. In RHEL 5.1 or later use the divider=10 kernel boot parameter to reduce the timer interrupt rate to 100 Hz. NOTE A bug in the RHEL 5.1 x86_64 kernel causes problems with the divider option. For RHEL 5.1 use the patch that fixes the issue at https://bugzilla.redhat.com/show_bug.cgi?id=305011. This bug is also fixed in RHEL 5.2. For more information see http://rhn.redhat.com/errata/RHSA-2007-0993.html. Kernels with tickless-timer support (NO_HZ kernels) do not schedule periodic timers to maintain system time. As a result, these kernels reduce the overall average rate of virtual timer interrupts, thus improving system performance and scalability on hosts running large numbers of virtual machines. Use a VMI-enabled operating system and enable VMI for the virtual machine (see Running Paravirtualized Operating Systems on page 29). For background information on this topic refer to Timekeeping in Virtual Machines, listed in Related Publications on page 8.

VMware Distributed Resource Scheduler (DRS) Best Practices
Clustering configurations can have a significant impact on performance. This section lists Distributed Resource Scheduler (DRS) practices and configurations recommended by VMware for optimal performance. When deciding which hosts to group into DRS clusters, try to choose hosts that are as homogeneous as possible in terms of CPU and memory. This ensures higher performance predictability and stability. VMware VMotion is not supported across hosts with incompatible CPU's. Hence with incompatible CPU heterogeneous systems, the opportunities DRS has to improve the load balance across the cluster are limited. To ensure CPU compatibility make sure systems are configured with the same CPU vendor, with similar CPU families, and with matching SSE instruction-set capability. For more information on this topic see KB articles 1991, 1992, and 1993, listed in Related Publications on page 8. You can also use Enhanced VMotion Compatibility (EVC) to facilitate VMotion between different CPU generations. For more information on this topic see VMware VMotion and CPU Compatibility, listed in Related Publications on page 8. When heterogeneous systems have compatible CPUs, but have different CPU frequencies and/or amounts of memory, DRS generally prefers to locate virtual machines on the systems with more memory and higher CPU frequencies (all other things being equal) since those systems have more capacity to accommodate peak load. The more VMotion compatible ESX hosts DRS has available, the more choices it has to better balance the DRS cluster. Besides CPU incompatibility, there are other misconfigurations that can block VMotion between two or more hosts. For example, if the hosts' VMotion network adapters are not connected by a Gigabit Ethernet link then the VMotion might not occur between the hosts. Other configuration settings to check for are virtual hardware version compatibility, misconfiguration of the VMotion gateway, incompatible security policies between the source and destination host VMotion network adapter, and virtual machine network availability on the destination host. Refer to the Resource Management Guide and the Server Configuration Guide for further details. Dont configure more hosts in a DRS cluster than the maximum allowed in the Configuration Maximums for VMware vSphere 4, listed in Related Publications on page 8. Virtual machines with smaller memory sizes and/or fewer vCPUs provide more opportunities for DRS to migrate them in order to improve balance across the cluster. Virtual machines with larger memory size and/or more vCPUs add more constraints in migrating the virtual machines. This is one more reason to configure virtual machines with only as many vCPUs and only as much virtual memory as they need. Have virtual machines in DRS automatic mode when possible, as they are considered for cluster load balance migrations across the ESX hosts before the virtual machines that are not in automatic mode. All powered-on virtual machines consume CPU resources. Thus even idle virtual machines, though their CPU utilization is usually small, can affect DRS decisions. For this and other reasons, a marginal performance increase might be obtained by shutting down (or suspending) virtual machines that are not being used. The migration threshold should be set to more aggressive levels when the following conditions are satisfied: If the hosts in the cluster are relatively homogeneous If the virtual machines' resource utilization does not vary too much over time and you have relatively few constraints on where a virtual machine can be placed The migration threshold should be set to more conservative levels in the converse situations. The frequency with which the DRS algorithm is invoked for balancing can be controlled through the vpxd configuration file, vpxd.cfg, with the following option:

VMware Distributed Power Management (DPM) Best Practices
Distributed Power Management (DPM) conserves power by migrating virtual machines to fewer hosts when utilizations are low. DPM is most appropriate for clusters in which composite virtual machine demand varies greatly over time; for example, clusters in which overall demand is higher during the day and significantly lower at night. If demand is consistently high relative to overall cluster capacity DPM will have little opportunity to put hosts into standby mode to save power. Most DRS best practices (described in VMware Distributed Resource Scheduler (DRS) Best Practices on page 38) are relevant to DPM as well. The more hosts in the cluster, the more opportunity there is for power savings through DPM. However dont exceed the maximum allowed in the Configuration Maximums for VMware vSphere 4, listed in Related Publications on page 8. DPM can only be enabled on hosts running ESX 3.5 or later. (Note that DPM is supported between hosts running EX 3.5 and ESX 4.0, subject to the VMotion limitations detailed in VMware VMotion and Storage VMotion on page 37.) Having VMotion-compatible hosts in the cluster is important, since DPM needs to be able to migrate running virtual machines onto fewer hosts to be able to put some hosts into standby mode to save power. Similarly, having wake-on-LAN (WOL) capable hosts in the cluster is beneficial. The ability of DPM to place a host in standby and power it on again later depends on ESX host software and on the network interface having hardware support for wake-on-lan. The more of the hosts in the cluster support DPM standby/power-on operations the more choices DPM has for saving power. Enabling DPM with all hosts in the cluster being in automatic mode gives DPM the most flexibility in choosing hosts for power down/up. For hosts in manual DPM mode, DPM must query the user about power down/up of the host. For this reason, DPM prefers choosing hosts in automatic mode over those in manual mode. If desired, DPM can be disabled for specific hosts. DPM considers historical demand in determining how much capacity to keep powered on and keeps some excess capacity available for changes in demand. DRS will also power on additional hosts when needed for unexpected increases in the demand of existing virtual machines or to allow virtual machine admission. In a cluster with VMware High Availability (HA) enabled, DRS/DPM maintains excess powered-on capacity to meet the High Availability settings. The cluster may therefore not allow additional virtual machines to be powered on and/or some hosts may not be powered down even though the cluster may appear to be sufficiently idle. These factors should be considered when configuring HA.

Wake-on-LAN A feature allowing a computer system to be powered on or brought out of suspend by sending a command over Ethernet.

Numerics

10 Gigabit Ethernet and NetQueue 14 64-bit DMA addresses 14
CPU affinity and hyper-threading 20 CPU virtualization hardware-assisted 11
active/active storage arrays policy 27 active/passive storage arrays policy 27 alignment file system partitions 26 AMD PCnet32 device 33 AMD PowerNow! 15 AMD-V 15, 21 anti-virus programs scheduling 29
disks eager-zeroed 26 independent nonpersistent 26 independent persistent 26 lazy-zeroed 26 snapshot 26 thick 26 thin 26 Distributed Power Management See DPM Distributed Resource Scheduler See DRS DPM (Distributed Power Management) 40 and reserve capacity 40 automatic vs. manual mode 40 DRS (Distributed Resource Scheduler) 38 affinity 39 algorithm frequency 38 and limits 39 and reservations 39 and shares 39 DVD drives 17
backups scheduling 27, 29 balloon driver and VMware Tools 29 ballooning 23 binary translation 21 BIOS settings 15 bridge chip 14 BT (binary translation) 21 bus architecture PCI 14 PCI Express 14 PCIe 14 PCI-X 14 BusLogic virtual storage adapter 32 using custom driver 32
E1000 device 33 eager-zeroed disks 26 Enhanced AMD PowerNow! 15 Enhanced Intel SpeedStep 15 Enhanced VMotion Compatibility 38 Enhanced VMXNET 33 EPT (extended page tables) 11, 15 esxtop and resxtop 19, 27 Ethernet 10 Gigabit
C1E halt state 15 CD drives 17 checksum offload 14 COM ports 17 CPU compatibility 11 overhead 19 reservations 19

and NetQueue

and iSCSI 13 and NFS 13 EVC (Enhanced VMotion Compatibility) 38 extended page tables 11, 15

Fault Tolerance 22, 41

file system partitions alignment 26 Floppy drives 17 FT (Fault Tolerance) 41
HA (High Availability) 40 HAL UP vs. SMP 20 hardware BIOS settings 15 minimum configuration 11 hardware compatibility list 11 hardware version 7 18, 33 and VMotion 37 hardware virtualization 21 hardware-assisted CPU virtualization 21 hardware-assisted MMU virtualization 11, 21 configuring ESX for 22 HBAs multiple 13 High Availability 40 high-memory DMA 14 HV (hardware virtualization) 21 hyper-threading 15, 20 CPU numbers 20
lazy-zeroed disks 26 limits and DRS 39 LPT ports 17 LSILogic virtual storage adapter 32
memory ballooning 23 large pages 25 overcommitment 24 overhead 23 page sharing 23 reclamation 24 reservation 24 sizing 23 swapping 23, 24 testing 11 memory management unit virtualization 11 hardware-assisted 11 memory swapping using reservations to avoid 24 MMU virtualization 11 hardware-assisted 11 MTU size 33
NAS network protocol processing 13 Nehalem 21 nested page tables 11 NetQueue 14 network throughput and CPU utilization 28 NFS and Ethernet 13 NIC team 14 NICs autonegotiation 14 duplex 14 server class 14 speed 14 NO_HZ kernels 30, 31 node interleaving 15, 21 non-uniform memory access 21 NPT (nested page tables) 11 NTP 29 NUMA (non-uniform memory access) 21

I/O block sizes 32 IBM X-Architecture 21 independent nonpersistent virtual disks 26 independent persistent virtual disks 26 Intel E1000 device 33 Intel Nehalem 21 Intel SpeedStep 15 Intel VT-x 15, 21 iSCSI and Ethernet 13 software-initiated
network protocol processing

ISO images 17

jumbo frames 14, 33
kernel UP vs. SMP 20 knowledge base accessing 8
Optical drives 17 overhead CPU 19

large pages 25

page sharing 23 paravirtual timer 30 paravirtualization 29 PCI bus architecture 14 PCI Express bus architecture 14 PCIe bus architecture 14 PCI-X bus architecture 14 PCnet32 device 33 PVSCSI virtual storage adapter 18, 32
SVGA frame buffer 23 swapping 23
TCP segmentation offload 14 thick disks 26 thin disks 26 tickless timer 30, 31 timekeeping 29 timing within virtual machines 30 TLB (translation lookaside buffer) 11 translation lookaside buffer 11 TSO 14, 33 Turbo Mode 15
queue depth 13 driver 27 virtual SCSI driver 32
Ubuntu and paravirtualization 30 Update Manager 36 USB controllers 17 user groups accessing 8
rapid virtualization indexing 11, 15 raw device mapping 26 RDM 26 receive buffers insufficient 33 reservation use of 35 reservations and DRS 39 resource pools 35, 39 RVI (rapid virtualization indexing) 11, 15
vCenter 36 alarm settings 36 database 36 statistics level 36 supported maximums 36 Update Manager 36 vCenter Converter 36 vCenter Management Webservices 36 vCPUs number of 19 virtual hardware version and VMotion 37 virtual machine interface 29 virtual machine monitor 21 virtual network adapter Evlance 33 VMXNET family 33 virtual SMP 19 vlance virtual network device 33 VLANs and storage 13 VMI 22, 29 VMI paravirtual timer 30 VMkernel network device driver 28 vmkfstools 26 VMM (virtual machine monitor) 21
shadow page tables 11 shares and DRS 39 use of 35 SLES and paravirtualization 30 SMP virtual 19 snapshot virtual disks 26 storage adapter BusLogic 32 LSILogic 32 PVSCSI 32 storage arrays active/active policy 27 active/passive policy 27 storage processors assigning 13 Storage VMotion 13, 37
VMotion and network adapters 38 compatible hosts and DPM 40 CPU compatibility 38 VMware community forums accessing 8 VMware Fault Tolerance 41 VMware Paravirtual storage adapter See PVSCSI VMware Storage VMotion 37 VMware Tools 32 and BusLogic SCSI driver 29 balloon driver 29 VMware vCenter 36 VMware vCenter Converter 36 VMware vCenter Management Webservices 36 VMware vSphere Client 36 VMXNET 33 VMXNET Generation 3 See VMXNET3 VMXNETVPNs and storage 13 vSphere Client 36 vSwitch 14 VT-x 15, 21

Configuration Maximums
VMware vSphere 4.0 and vSphere 4.0 Update 1
Whenyouselectandconfigureyourvirtualandphysicalequipment,youmuststayatorbelowthemaximums supportedbyvSphere4.0andvSphere4.0Update1.Thelimitspresentedinthefollowingtablesrepresent tested,recommendedlimits,andtheyarefullysupportedbyVMware.
VirtualMachineMaximumsonpage 1 ESXHostMaximumsonpage 3 vCenterServerMaximumsonpage 7 vCenterServerExtensionsonpage 8
Thelimitspresentedinthisdocumentcanbeaffectedbyotherfactors,suchashardwaredependencies.For moreinformationaboutsupportedhardware,seetheappropriateESXhardwarecompatibilityguide.Please consultindividualsolutionlimitstoensurethatyoudonotexceedsupportedconfigurationsforyour environment. TheConfigurationMaximumsforvSphere4.0andvSphere4.0Update1coversESX,ESXi,andvCenterServer.

Virtual Machine Maximums

Table 1containsconfigurationmaximumsrelatedtovirtualmachines. Table 1. Virtual Machine Maximums
Item Compute VirtualCPUspervirtualmachine(VirtualSMP) Memory RAMpervirtualmachine Virtualmachineswapfilesize Storage Virtual Adapters and Devices VirtualSCSIadapterspervirtualmachine VirtualSCSItargetspervirtualSCSIadapter VirtualSCSItargetspervirtualmachine Disksize IDEcontrollerspervirtualmachine IDEdevicespervirtualmachine Floppycontrollerspervirtualmachine 60 2TBminus 512B255GB 255GBMaximum

VMware, Inc.

Table 1. Virtual Machine Maximums (Continued)
Item Floppydevicespervirtualmachine Networking Virtual Devices VirtualNICspervirtualmachine Virtual Peripheral Ports Parallelportspervirtualmachine Serialportspervirtualmachine VMDirectPath VMDirectPathPCI/PCIedevicespervirtualmachine VMDirectPathSCSItargetspervirtualmachine Miscellaneous Concurrentremoteconsoleconnectionstoavirtualmachine
1. Sameasthemaximumvirtualmachinememorysize. 2. AnycombinationofsupportedSCSIvirtualstoragecontrollers.Four ParavirtualSCSIadaptersmaybeusedonlyifthevirtualmachineboots fromadeviceattachedtoanIDEcontroller,orfromthenetwork. 3. Anycombinationofdisk,CDROMorVMDirectPathSCSItarget. 4. LimitedbymaximumVMFSfilesize(assumes8MBVMFSblocksize). 5. Supportstwochannels(primaryandsecondary)eachwithamasterand slavedevice. 6. DevicescanbeeitherCDROMordisk. 7. BIOSisconfiguredforonefloppydevice. 8. AnycombinationofsupportedvirtualNICs. 9. RequiresI/OMMUonhost.

Maximum 27

ESX Host Maximums
ThefollowingtablescontainconfigurationmaximumsrelatedtoESXhosts.
StorageMaximumsonpage 3 ComputeMaximumsonpage 5 MemoryMaximumsonpage 5 NetworkingMaximumsonpage 6 ResourcePoolandClusterMaximumsonpage 7

Storage Maximums

VMFS2issupportedonESX3.0through3.5. Table 2containsconfigurationmaximumsrelatedtoESXhoststorage. Table 2. Storage Maximums
Item VMFS-General Rawdevicemapping(RDM)size Volumesize4 Virtualmachinespervolume Volumesperhost Extentspervolume Hostspercluster Extentssize MaxI/Osize(beforesplits) VMFS-2 Filespervolume 256+(64x additional extents) 256MB 456GB 2TB 27TB 64TB 2TBminus 512B 64TBminus 16K 2TBminus 512B 32MB Maximum
Blocksize Filesize(blocksize=1MB) Filesize(blocksize=8MB) Filesize(blocksize=64MB) Filesize(blocksize=256MB) VMFS-3 VMFS3volumesconfiguredperhost Filespervolume Blocksize Filesize(blocksize=1MB) Filesize(blocksize=2MB) Filesize(blocksize=4MB)
256 ~30,7201 8MB 256GBminus 512B 512GBminus 512B 1TBminus 512B
Table 2. Storage Maximums (Continued)
Item Filesize(blocksize=8MB) NFS DefaultNFSdatastores NFSdatastores Fibre Channel LUNsperhost LUNsize PathstoaLUN Totalpathsonahost LUNsconcurrentlyopenedbyallvirtualmachines LUNID HBAsperhost HBAports TargetsperHBA Hardware iSCSI Initiators LUNsperhost LUNsconcurrentlyused Initiatorportsperhost Totalpathsonahost PathstoaLUN Dynamictargetsperadapterport Statictargetsperadapterport Software iSCSI Initiators LUNsperhost LUNsconcurrentlyused NICsportboundwiththesoftwareiSCSIstackperserver Targets(thesumofstatictargetsanddynamictargetsmaynotexceed thisfigure) PathstoaLUN Totalpaths
1. Sufficienttosupportthemaximumvirtualmachines.

Maximum 2TBminus 512B

2563 2TBminus 512B 256

8 1024

2. Requireschangestoadvancedsettings. 3. Includeslocaldevices/disks.
4. VolumeisnotthesameasLUN.InordertoexceedtheLUNsize,youmustuseextents.

Compute Maximums

Table 3containsconfigurationmaximumsrelatedtoESXhostcomputeresources. Table 3. Compute Maximums
Item VirtualCPUsperhost Virtualmachinesperhost Logicalprocessorsperhost VirtualCPUsperphysicalcore VirtualCPUsperphysicalcoreforvSphere4.0Update1
1. Specificsolutionlimitsmaybelower;pleasecheckindividualsolution limitsformaximumsupportedconfigurations. 2. LogicalCPUsperhost=CPUsocketsxcores/socketxthreads/core. RegardlessofthehostsconfigurationofCPUsockets,cores/socketor threadsperCPUcore,thetotalnumberoflogicalCPUs(hardware threads)maynotexceedthisnumber.LogicalCPUsinexcessofthis numberareignored. 3. TheachievablenumberofvCPUspercoredependsontheworkloadand specificsofthehardware.FormoreinformationseethePerformanceBest PracticesforVMwarevSphere4.0.

Maximum 253

Memory Maximums
Table 4containsconfigurationmaximumsrelatedtoESXhostmemory. Table 4. Memory Maximums
Item SizeofRAMperhost MaximumRAMallocatedtoserviceconsole MinimumRAMallocatedtoserviceconsole Swapfiles Swapfilesize
Maximum 1TB 800MB 300MB 1pervirtual machine Sameasmax virtual machineRAM

Networking Maximums

Thefollowinglimitsrepresentachievablemaximumconfigurationlimitsfornetworkinginenvironments wherenoothermorerestrictivelimitsapply.Forexample,vCenterServerlimits,thelimitsimposedby featuressuchasHAorDRS,andotherconfigurationsthatmightimposerestrictionsmustbeconsideredwhen deployinglargescalesystems. Foradditionalandrevisednumberstothesemaximums,seeKB1020808. Table 5containsconfigurationmaximumsrelatedtoESXhostnetworking. Table 5. Networking Maximums

Item Physical NICs

Maximum
e1000NICsEthernetports(IntelPCIxNIC) e1000eNICsEthernetports(IntelPCIeNIC) igb1GBEthernetports(Intel) tg31GBEthernetports(Broadcom) bnx21GBEthernetports(Broadcom) forcedeth1GBEthernetports(Nvidia) s2io10GBEthernetports(Neterion) nx_nic10GBEthernetports(NetXen) ixgbeOplin10GBEthernetports(Intel) bnx2x10GBEthernetports(Broadcom) Infinibandports(refertoVMwareCommunitySupport) PCI VMDirectPath Devices2 PCIVMDirectPathdevicesperhost vNetwork Standard Switch Totalvirtualnetworkswitchportsperhost(vDSandvSSports) Virtualnetworkswitchportsperstandardswitch Portgroupsperstandardswitch Standardswitchesperhost vNetwork Distributed Switch Totalvirtualnetworkswitchportsperhost(vDSandvSSports) DistributedvirtualnetworkswitchportspervCenter DistributedportgroupspervCenter DistributedswitchespervCenter Hostsperdistributedswitch
1. MellanoxTechnologiesInfiniBandHCAdevicedriversareavailable directlyfromMellanoxTechnologies.PleaserefertoMellanoxfor supportstatusofInfiniBandHCAswithESX. http://www.mellanox.com 2. Theselimitsaresupportedwithstandardswitchesanddistributed virtualswitches.

512 248

Resource Pool and Cluster Maximums
Table 6containsconfigurationmaximumsrelatedtoESXhostresourcepools. Table 6. Resource Pool Maximums
Item HA Cluster HostsperHAcluster VirtualmachinesperhostinHAclusterwith8orfewerhosts VirtualmachinesperhostinHAclusterwith8orfewerhostsfor vSphere4.0Update1 VirtualmachinesperhostinHAclusterwith9ormorehosts Failoverhostspercluster Failoveraspercentageofcluster DRS Cluster HostsperDRScluster VirtualmachinesperDRScluster VirtualmachinesperhostinDRScluster Resource Pool Resourcepoolsperhost Childrenperresourcepool Treedepthperresourcepool TreedepthperresourcepoolinaDRScluster Resourcepoolspercluster

VMware vCenter Orchestrator
Table 9containsconfigurationmaximumsforvCenterOrchestrator. Table 9. vCenter Orchestrator Maximums
Item ConnectedvCenterServersystems ConnectedESX/ESXiservers Connectedvirtualmachines Concurrentrunningworkflows Maximum 3000 150

VMware vCenter Converter

Table 10containsconfigurationmaximumsforvCenterConverter. Table 10. vCenter Converter Maximums
Item Concurrentimport/exporttasks(assumesnoloadonvCenterServer system) Maximum 16
vSphere Storage Management Initiative - Specification (SMI-S)
Table 11containsconfigurationmaximumsforvSphereSMIS. Table 11. vSphere SMI-S Maximums
Item NumberofvCenterServersystemsconnected NumberofESX/ESXihostsconnected NumberofESX/ESXihostsmanagedinvCenterServer NumberofvirtualmachinesregisteredinvCenterServer NumberofdatastoresregisteredinvCenterServer Maximum 100
If you have comments about this documentation, submit your feedback to: docfeedback@vmware.com VMware, Inc. 3401 Hillview Ave., Palo Alto, CA 94304 www.vmware.com Copyright 2006-2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies. Item: EN-000103-00