Tutorial: Profiling in Xen

J. Renato Santos

HP Labs
Xen Summit September 7-8, 2006
2003 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice
Preparing the environment

Xen version requirement

latest xen-unstable or xen 3.0.3 and above
Download oprofile 0.9.1 source (oprofile-0.9.1.tar.gz) from http://oprofile.sourceforge.net Download oprofile patch for Xen (oprofile-0.9.1-xen-rc2.patch) from http://xenoprof.sourceforge.net Compile and install oprofile modified for xen (as root)
> tar zxvf oprofile-0.9.1.tar.gz > cd oprofile-0.9.1 > patch p1 <./oprofile-0.9.1-xen-rc2.patch >./configure with-kernel-support > make > make install

September 11, 2006

page 2
Preparing the environment (cont)
Verify if we are using the right OProfile
opcontrol help If successful, above command should list --xen as a valid option
Edit.config file for dom0 and domU to enable OProfile module (do not use kernel builtin option)
CONFIG_PROFILING=y CONFIG_OPROFILE=m
Re-compile and re-install Xen in dom0 file system Copy Xen and kernel images (in /boot) to domU file system Copy domU modules (in /lib/modules) to domU file system We are now ready to start profiling

page 3

Active domain profiling (1)
Example: Profiling domain 0, domain 1, and domain 3
Remove any samples from previous runs (in dom0 and domUs)
dom0> opcontrol --reset dom1> opcontrol reset dom3> opcontrol --reset
Define profiling session parameters in dom0
dom0> opcontrol --start-daemon --event=GLOBAL_POWER_EVENTS:1000000 --xen=/boot/xen-syms-3.0-unstable vmlinux=/boot/vmlinux-syms-2.6.16.13-xen0 --active-domains=1,3 --start-daemon : start oprofile daemon with given parameters but do not start profiling yet. --event : event used for profiling (non-halted clock cycles, cache misses, etc). can be specified multiple times --xen : Xen image (uncompressed version) --vmlinux : kernel image for dom0 (uncompressed version) --active-domains : list of domain ids to be profiled in active mode

page 4

Active domain profiling (2)
Start OProfile in all active domUs
dom1> opcontrol --start --event=GLOBAL_POWER_EVENT:1000000:1:1:1 --xen=/boot/xen-syms-3.0-unstable vmlinux=/boot/vmlinux-syms-2.6.16.13-xenU dom3> opcontrol --start --event=GLOBAL_POWER_EVENT:1000000:1:1:1 --xen=/boot/xen-syms-3.0-unstable vmlinux=/boot/vmlinux-syms-2.6.16.13-xenU --event : need to specify same event as specified in dom0.
Start profiling (in dom0)
dom0> opcontrol --start
Run experiment to be profiled Stop profiling
dom0> opcontrol --stop

Shutdown OProfile daemon

dom0> opcontrol --shutdown dom1> opcontrol --shutdown dom3> opcontrol --shutdown

page 5

Passive domain profiling
Remove any samples from previous runs (dom0)
dom0> opcontrol --reset
Define profiling session parameters (dom0)
dom0> opcontrol --start-daemon --event=GLOBAL_POWER_EVENTS:1000000 --xen=/boot/xen-syms-3.0-unstable --vmlinux=/boot/vmlinux-syms-2.6.16.13-xen0 --passive-domains=1,3 --passive-images=/boot/vmlinux-syms-2.6.16.13-xenU,/boot/vmlinux-syms-2.6.16.13-xenU --passive-images : kernel images for passive domains (same order as --passive-domains)
dom0> opcontrol --shutdown
September 11, 2006 page 6
Obtaining profile results (1)
Command to get per binary image profile

> opreport

CPU: P4 / Xeon with 2 hyper-threads, speed 2794.57 MHz (estimated) Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 1000000 GLOBAL_POWER_E.| samples| %| -----------------82499 67.9306 vmlinux-syms-2.6.16.13-xenU 38297 31.5342 xen-syms-3.0-unstable 435 0.3582 rcv 93 0.0766 oprofiled 41 0.0338 libcrypto.so.0.9.7f 39 0.0321 libc-2.3.6.so 23 0.0189 ld-2.3.6.so 7 0.0058 bash 4 0.0033 nifd 3 0.0025 sshd

page 7

Obtaining profile results (2)
Command to get per symbol (function) profile

> opreport -l

CPU: P4 / Xeon with 2 hyper-threads, speed 2794.57 MHz (estimated) Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 1000000 samples % app name symbol name 6913 5.4236 xen-syms-3.0-unstable do_grant_table_op 5922 4.6461 vmlinux-syms-2.6.16-xen0 net_tx_action 5419 4.2515 xen-syms-3.0-unstable find_domain_by_id 5066 3.9745 xen-syms-3.0-unstable gnttab_transfer 4870 3.8208 vmlinux-syms-2.6.16-xen0 net_rx_action 4311 3.3822 vmlinux-syms-2.6.16-xen0 nf_iterate 4032 3.1633 vmlinux-syms-2.6.16-xen0 hypercall_page 3555 2.7891 vmlinux-syms-2.6.16-xen0 eth_type_trans 3397 2.6651 xen-syms-3.0-unstable hypercall 2820 2.2124 vmlinux-syms-2.6.16-xen0 e1000_intr 2607 2.0453 xen-syms-3.0-unstable alloc_domheap_pages 2549 1.9998 vmlinux-syms-2.6.16-xen0 nf_hook_slow 2264 1.7762 xen-syms-3.0-unstable evtchn_set_pending 2249 1.7645 vmlinux-syms-2.6.16-xen0 e1000_clean_rx_irq 2216 1.7386 xen-syms-3.0-unstable __copy_from_user_ll 2178 1.7088 vmlinux-syms-2.6.16-xen0 dev_queue_xmit 2160 1.6946 xen-syms-3.0-unstable get_page_type

0.0345 0.0345 0.0330 0.0322 0.0298 0.0290 0.0290
libc-2.3.6.so vmlinux-syms-2.6.16-xen0 vmlinux-syms-2.6.16-xen0 vmlinux-syms-2.6.16-xen0 xen-syms-3.0-unstable vmlinux-syms-2.6.16-xen0 vmlinux-syms-2.6.16-xen0
malloc e1000_update_stats dma_map_page raise_softirq_irqoff do_arch_sched_op ktime_get rcu_pending

page 8

Obtaining profile results (3)

Active domains

Multiple reports: one per active domain Passive domain samples reported in domain 0 report

Passive domains

CPU: P4 / Xeon with 2 hyper-threads, speed 2794.57 MHz (estimated) Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 100000 samples % image name app name symbol name 30353 12.0434 domain1-kernel domain1-kernel __ copy_to_user_ll 7174 2.8465 xen-syms-3.0-unstable xen-syms-3.0-unstable do_grant_table_op 6040 2.3965 vmlinux-syms-2.6.16.13-xen0 vmlinux-syms-2.6.16.13-xen0 net_tx_action 5508 2.1854 xen-syms-3.0-unstable xen-syms-3.0-unstable find_domain_by_id 4944 1.9617 xen-syms-3.0-unstable xen-syms-3.0-unstable gnttab_transfer 4848 1.9236 domain1-xen domain1-xen evtchn_set_pending 4631 1.8375 vmlinux-syms-2.6.16.13-xen0 vmlinux-syms-2.6.16.13-xen0 net_rx_action 4322 1.7149 domain1-kernel domain1-kernel tcp_v4_rcv 4145 1.6446 xen-syms-3.0-unstable xen-syms-3.0-unstable hypercall 4005 1.5891 domain1-xen domain1-xen guest_remove_page 3644 1.4459 vmlinux-syms-2.6.16.13-xen0 vmlinux-syms-2.6.16.13-xen0 hypercall_page 3589 1.4240 vmlinux-syms-2.6.16.13-xen0 vmlinux-syms-2.6.16.13-xen0 eth_type_trans 3425 1.3590 domain1-xen domain1-xen get_page_from_l1e 2846 1.1292 domain1-kernel domain1-kernel eth_type_trans 2770 1.0991 vmlinux-syms-2.6.16.13-xen0 vmlinux-syms-2.6.16.13-xen0 e1000_intr
0.0298 0.0274 0.0274 0.0270 0.0262
domain1-apps domain1-kernel vmlinux-syms-2.6.16.13-xen0 domain1-kernel oprofiled
(no symbols) ns_to_timespec delay_tsc do_IRQ odb_insert

page 9

Other useful OProfile commands
List hardware performance events for current CPU model
opcontrol -l Opcontrol deinit
Unload oprofile module More information
OProfile user manual (http://oprofile.sourceforge.net) OProfile man pages XenOprofile user guide (http://xenoprof.sourceforge.net)

page 10

HP logo

page 11

How Does Xen Work?

Date: Version:

December 2009 1.0
Table of Contents Executive Summary....3 Xen Environment Components....3 Xen Hypervisor....3 Domain 0....4 Domain U....4 Domain Management and Control...6 Xend.....6 Xm.....6 Xenstored....6 Libxenctrl....6 Qemu-dm....7 Xen Virtual Firmware....7 Xen Operation.....7 Domain 0 to Domain U Communication...8 Xen PCI Passthru.....9 Glossary....9

How Does Xen Work

Executive Summary
This document contains a high level architectural overview of the Xen hypervisor and the associated tools and applications necessary to support a complete virtualization environment. The architecture is based on Xen 3.4 released in late 2009 and is only an introduction to the overall Xen architecture. For a more complete description of the architecture please reference the many Xen books available.
Xen Environment Components
A Xen virtual environment consist of several items that work together to deliver the virtualization environment a customer is looking to deploy: Xen Hypervisor Domain 0 Guest o Domain Management and Control (Xen DM&C) Domain U Guest (Dom U) o PV Guest o HVM Guest
The diagram below shows the basic organization of these components.

Domain 0 Guest

Domain Guest

Hardware

Xen Hypervisor
The Xen hypervisor is the basic abstraction layer of software that sits directly on the hardware below any operating systems. It is responsible for CPU scheduling and memory partitioning of the various virtual machines running on the hardware device. The hypervisor not only abstracts the hardware for the virtual machines but also controls the execution of virtual machines as they share the common processing environment. It has no knowledge of networking, external storage devices, video, or any other common I/O functions found on a computing system. How Does Xen Work 3

Domain 0

Domain 0, a modified Linux kernel, is a unique virtual machine running on the Xen hypervisor that has special rights to access physical I/O resources as well as interact with the other virtual machines (Domain U: PV and HVM Guests) running on the system. All Xen virtualization environments require Domain 0 to be running before any other virtual machines can be started. Two drivers are included in Domain 0 to support network and local disk requests from Domain U PV and HVM Guests (see below); the Network Backend Driver and the Block Backend Driver. The Network Backend Driver communicates directly with the local networking hardware to process all virtual machines requests coming from the Domain U guests. The Block Backend Driver communicates with the local storage disk to read and write data from the drive based upon Domain U requests.
PV Drivers: NetworkDomain 0 Driver Backend

Block Backend Driver

Domain U
DomainU guests have no direct access to physical hardware on the machine as a Domain0 Guest does and is often referred to as unprivileged. All paravirtualized virtual machines running on a Xen hypervisor are referred to as Domain U PV Guests and are modified Linux operating systems, Solaris, FreeBSD, and other UNIX operating systems. All fully virtualized machines running on a Xen hypervisor are referred to as Domain U HVM Guests and run standard Windows or any other unchanged operating system. The Domain U PV Guest virtual machine is aware that it does not have direct access to the hardware and recognizes that other virtual machines are running on the same machine. The Domain U HVM Guest virtual machine is not aware that it is sharing processing time on the hardware and that other virtual machines are present. A Domain U PV Guest contains two drivers for network and disk access, PV Network Driver and PV Block Driver. How Does Xen Work 4

Domain U PV Guest PV Drivers: Network Driver

Block Driver

A Domain U HVM Guest does not have the PV drivers located within the virtual machine; instead a special daemon is started for each HVM Guest in Domain 0, Qemudm. Qemu-dm supports the Domain U HVM Guest for networking and disk access requests. The Domain U HVM Guest must initialize as it would on a typical machine so software is added to the Domain U HVM Guest, Xen virtual firmware, to simulate the BIOS an operating system would expect on startup. More information on the Xen virtual firmware is presented later in this document.

Domain 0 Qemu-DM

Domain U HVM Guest Xen virtual firmware
Domain Management and Control
A series of Linux daemons are classified as Domain Management and Control by the open source community. These services support the overall management and control of the virtualization environment and exist within the Domain 0 virtual machine. The diagrams below show the daemons outside the Domain 0 diagram for a clearer understanding of the architecture.
The Xend daemon is a python application that is considered the system manager for the Xen environment. It leverages the libxenctrl library (see below) to make requests of the Xen hypervisor. All requests processed by the Xend are delivered to it via an XML RPC interface by the Xm (see below) tool.

Xend Libxenctrl Domain 0

XML RPC
The command line tool that takes user input and passes to Xend via XML RPC.

Xenstored

The Xenstored daemon maintains a registry of information including memory and event channel links between Domain 0 and all other Domain U Guests. The Domain 0 virtual machine leverages this registry to setup device channels with other virtual machines on the system. (See Domain 0 to Domain U Communication for more details).

Libxenctrl

Libxenctrl is a C library that provides Xend the ability to talk with the Xen hypervisor via Domain 0. A special driver within Domain 0, privcmd delivers the request to the hypervisor. How Does Xen Work 6

Xend Domain 0 Libxenctrl

privcmd

Qemu-dm

Every HVM Guest running on a Xen environment requires its own Qemu daemon. This tool handles all networking and disk requests from the Domain U HVM Guest to allow for a fully virtualized machine in the Xen environment. Qemu must exist outside the Xen hypervisor due to its need for access to networking and I/O and is therefore found in Domain 0. A new tool, Stub-dm, is in development for future versions of Xen that will remove the need for a Qemu running for every Domain U HVM Guest and will instead provide a set of services available to every Domain U HVM Guest. This feature is not available in Xen 3.2 but is currently part of the xen-unstable tree and will be released as part of Xen 3.3.

Xen Virtual Firmware

The Xen Virtual Firmware is a virtual BIOS that is inserted into every Domain U HVM Guest to ensure that the operating system receives all the standard start-up instructions it expects during normal boot-up providing a standard PC-compatible software environment.

Xen Operation

This section demonstrates how a paravirtualized Domain U is able to communicate with external networks or storage via the Xen hypervisor and Domain 0.
Domain 0 to Domain U Communication
As stated earlier, the Xen hypervisor is not written to support network or disk requests thus a Domain U PV Guest must communicate via the Xen hypervisor with the Domain 0 to accomplish a network or disk request. The example presented below shows a Domain U PV Guest writing data to the local hard disk. The Domain U PV Guest PV block driver receives a request to write to the local disk and writes the data via the Xen hypervisor to the appropriate local memory which is shared with Domain 0. An event channel exists between Domain 0 and the Domain U PV Guest that allows them to communicate via asynchronous inter-domain interrupts in the Xen hypervisor. Domain 0 will receive an interrupt from the Xen hypervisor causing the PV Block Backend Driver to access the local system memory reading the appropriate blocks from the Domain U PV Guest shared memory. The data from shared memory is then written to the local hard disk at a specific location. The event channel is shown below as a direct link between Domain 0 and Domain U PV Guest which is a simplified view of the way the system works. In fact, the event channel runs through the Xen hypervisor with specific interrupts registered in Xenstored allowing both the Domain 0 and Domain U PV Guest to quickly share information across local memory. The figure below shows the previously described situation:
Domain 0 PV Block Backend Driver

Event Channel

Domain U PV Guest PV Block Driver
Xen Hypervisor Shared Memory Data Dom U Data Dom U

Xen PCI Passthru

A new feature in Xen designed to improve overall performance and reduce the load on the Dom 0 Guest is PCI Passthru which allows the Domain U Guest to have direct access to local hardware without using the Domain 0 for hardware access. The diagram below shows how this feature works:

Hardware Hardware Device

The Domain U Guest is given rights to talk directly to a specific hardware device instead of the previous method of using Fronted and Backend drivers.

Glossary

C: http://www.cprogramming.com/; a computer programming language Daemons: http://en.wikipedia.org/wiki/Daemon_(computer_software); a program running in the background rather than under direct control of a user Driver: http://en.wikipedia.org/wiki/Device_driver; program allowing software to interact with hardware Full Virtualization: http://en.wikipedia.org/wiki/Full_virtualization; a virtual machine not aware of its virtualization

Interrupt: http://en.wikipedia.org/wiki/Interrupt; signal from hardware to software requesting a specific action in software Kernel: http://en.wikipedia.org/wiki/Linux_kernel; the central component of a computer operating system Paravirtualized: http://en.wikipedia.org/wiki/Paravirtualization; virtual machine running on a hypervisor that is aware of it being virtualized] Python: http://www.python.org/; a dynamic object oriented programming language ROM BIOS: http://en.wikipedia.org/wiki/BIOS; software instructions run on a machine when turned on XML PRC: http://www.xmlrpc.com/; method for an application to leverage another application using HTTP for the remote procedure call and XML as the encoding