High-level architecture

The XAPI Toolstack manages a cluster of hosts, network switches and storage on behalf of clients such as XenCenter and Xen Orchestra.

The most fundamental concept is of a Resource pool: the whole cluster managed as a single entity. The following diagram shows a cluster of hosts running xapi, all sharing some storage:

A Resource Pool A Resource Pool

At any time, at most one host is known as the pool coordinator (formerly known as “master”) and is responsible for coordination and locking resources within the pool. When a pool is first created a coordinator host is chosen. The coordinator role can be transferred

  • on user request in an orderly fashion (xe pool-designate-new-master)
  • on user request in an emergency (xe pool-emergency-transition-to-master)
  • automatically if HA is enabled on the cluster.

All hosts expose an HTTP, XML-RPC and JSON-RPC interface running on port 80 and with TLS on port 443, but control operations will only be processed on the coordinator host. Attempts to send a control operation to another host will result in a XenAPI redirect error message. For efficiency the following operations are permitted on non-coordinator hosts:

  • querying performance counters (and their history)
  • connecting to VNC consoles
  • import/export (particularly when disks are on local storage)

Since the coordinator host acts as coordinator and lock manager, the other hosts will often talk to the coordinator. Non-coordinator hosts will also talk to each other (over the same HTTP and RPC channels) to

  • transfer VM memory images (VM migration)
  • mirror disks (storage migration)

Note that some types of shared storage (in particular all those using vhd) require coordination for disk GC and coalesce. This coordination is currently done by xapi and hence it is not possible to share this kind of storage between resource pools.

The following diagram shows the software running on a single host. Note that all hosts run the same software (although not necessarily the same version, if we are in the middle of a rolling update).

A Host A Host

The XAPI Toolstack expects the host to be running Xen on x86. The Xen hypervisor partitions the host into Domains, some of which can have privileged hardware access, and the rest are unprivileged guests. The XAPI Toolstack normally runs all of its components in the privileged initial domain, Domain 0, also known as “the control domain”. However there is experimental code which supports “driver domains” allowing storage and networking drivers to be isolated in their own domains.

Subsections of High-level architecture

Environment

The Toolstack runs in an environment on a server (host) that has:

  • Physical hardware.
  • The Xen hypervisor.
  • The control domain (domain 0): the priviledged domain that the Toolstack runs in.
  • Other, mostly unpriviledged domains, usually for guests (VMs).

The Toolstack relies on various bits of software inside the control domain, and directly communicates with most of these:

  • Linux kernel including drivers for hardware and Xen paravirtualised devices (e.g. netback and blkback).
    • Interacts through /sys and /proc, udev scripts, xenstore, …
  • CentOS distibution including userspace tools and libraries.
    • systemd, networking tools, …
  • Xen-specific libraries, especially libxenctrl (a.k.a. libxc)
  • xenstored: a key-value pair configuration database
    • Accessible from all domains on a host, which makes it useful for inter-domain communication.
    • The control domain has access to the entire xenstore database, while other domains only see sub-trees that are specific to that domain.
    • Used for connecting VM disks and network interfaces, and other VM configuration options.
    • Used for VM status reporting, e.g. the capabilities of the PV drivers (if installed), the IP address, etc.
  • SM: Storage Manager plugins which connect xapi’s internal storage interfaces to the control APIs of external storage systems.
  • stunnel: a daemon which decodes TLS and forwards traffic to xapi (and the other way around).
  • Open vSwitch (OVS): a virtual network switch, used to connect VMs to network interfaces. The OVS offers several networking features that xapi takes advantage of.
  • QEMU: emulation of various bits of hardware
  • DEMU: emulation of Nvidia vGPUs
  • xenguest
  • emu-manager
  • pvsproxy
  • xenconsoled: allows access to guest consoles. This is common to all Xen hosts.

The Toolstack also interacts with software that runs inside the guests:

  • PV drivers
  • The guest agent

Daemons

The Toolstack consists of a set of co-operating daemons:

xapi
manages clusters of hosts, co-ordinating access to shared storage and networking.
xenopsd
a low-level “domain manager” which takes care of creating, suspending, resuming, migrating, rebooting domains by interacting with Xen via libxc and libxl.
xcp-rrdd
a performance counter monitoring daemon which aggregates “datasources” defined via a plugin API and records history for each. There are various rrdd-plugin daemons:
  • xcp-rrdd-gpumon
  • xcp-rrdd-iostat
  • xcp-rrdd-squeezed
  • xcp-rrdd-xenpm
xcp-networkd
a host network manager which takes care of configuring interfaces, bridges and OpenVSwitch instances
squeezed
a daemon in charge of VM memory management
xapi-storage-script
for storage manipulation over SMAPIv3
message-switch
exchanges messages between the daemons on a host
xapi-guard
forwards uefi and vtpm persistence calls from domains to xapi
v6d
controls which features are enabled.
forkexecd
a helper daemon that assists the above daemons with executing binaries and scripts
xhad
The High-Availability daemon
perfmon
a daemon which monitors performance counters and sends “alerts” if values exceed some pre-defined threshold
mpathalert
a daemon which monitors “storage paths” and sends “alerts” if paths fail and need repair
wsproxy
handles access to VM consoles

Interfaces

Communication between the Toolstack daemon is built upon libraries from a component called xapi-idl.

  • Abstracts communication between daemons over the message-switch using JSON/RPC.
  • Contains the definition of the interfaces exposed by the daemons (except xapi).