Improving QEMU security part 7: TLS support for migration

Posted: August 16th, 2016 | Author: | Filed under: Coding Tips, Fedora, libvirt, OpenStack, Security, Virt Tools | Tags: , , | No Comments »

This blog is part 7 of a series I am writing about work I’ve completed over the past few releases to improve QEMU security related features.

The live migration feature in QEMU allows a running VM to be moved from one host to another with no noticeable interruption in service and minimal performance impact. The live migration data stream will contain a serialized copy of state of all emulated devices, along with all the guest RAM. In some versions of QEMU it is also used to transfer disk image content, but in modern QEMU use of the NBD protocol is preferred for this purpose. The guest RAM in particular can contain sensitive data that needs to be protected against any would be attackers on the network between source and target hosts. There are a number of ways to provide such security using external tools/services including VPNs, IPsec, SSH/stunnel tunnelling. The libvirtd daemon often already has a secure connection between the source and destination hosts for its own purposes, so many years back support was added to libvirt to automatically tunnel the live migration data stream over libvirt’s own secure connection. This solved both the encryption and authentication problems at once, but there are some downsides to this approach. Tunnelling the connection means extra data copies for the live migration traffic and when we look at guests with RAM many GB in size, the number of data copies will start to matter. The libvirt tunnel only supports a tunnelling of a single data connection and in future QEMU may well wish to use multiple TCP connections for the migration data stream to improve performance of post-copy. The use of NBD for storage migration is not supported with tunnelling via libvirt, since it would require extra connections too. IOW while tunnelling over libvirt was a useful short term hack to provide security, it has outlived its practicality.

It is clear that QEMU needs to support TLS encryption natively on its live migration connections. The QEMU migration code has historically had its own distinct I/O layer called QEMUFile which mixes up tracking of migration state with the connection establishment and I/O transfer support. As mentioned in previous blog post, QEMU now has a general purpose I/O channel framework, so the bulk of the work involved converting the migration code over to use the QIOChannel classes and APIs, which greatly reduced the amount of code in the QEMU migration/ sub-folder as well as simplifying it somewhat. The TLS support involves the addition of two new parameters to the migration code. First the “tls-creds” parameter provides the ID of a previously created TLS credential object, thus enabling use of TLS on the migration channel. This must be set on both the source and target QEMU’s involved in the migration.

On the target host, QEMU would be launched with a set of TLS credentials for a server endpoint:

$ qemu-system-x86_64 -monitor stdio -incoming defer \
    -object tls-creds-x509,dir=/home/berrange/security/qemutls,endpoint=server,id=tls0 \
    ...other args...

To enable incoming TLS migration 2 monitor commands are then used

(qemu) migrate_set_str_parameter tls-creds tls0
(qemu) migrate_incoming tcp:myhostname:9000

On the source host, QEMU is launched in a similar manner but using client endpoint credentials

$ qemu-system-x86_64 -monitor stdio \
    -object tls-creds-x509,dir=/home/berrange/security/qemutls,endpoint=client,id=tls0 \
    ...other args...

To enable outgoing TLS migration 2 monitor commands are then used

(qemu) migrate_set_str_parameter tls-creds tls0
(qemu) migrate tcp:otherhostname:9000

The migration code supports a number of different protocols besides just “tcp:“. In particular it allows an “fd:” protocol to tell QEMU to use a passed-in file descriptor, and an “exec:” protocol to tell QEMU to launch an external command to tunnel the connection. It is desirable to be able to use TLS with these protocols too, but when using TLS the client QEMU needs to know the hostname of the target QEMU in order to correctly validate the x509 certificate it receives. Thus, a second “tls-hostname” parameter was added to allow QEMU to be informed of the hostname to use for x509 certificate validation when using a non-tcp migration protocol. This can be set on the source QEMU prior to starting the migration using the “migrate_set_str_parameter” monitor command

(qemu) migrate_set_str_parameter tls-hostname myhost.mydomain

This feature has been under development for a while and finally merged into QEMU GIT early in the 2.7.0 development cycle, so will be available for use when 2.7.0 is released in a few weeks. With the arrival of the 2.7.0 release there will finally be TLS support across all QEMU host services where TCP connections are commonly used, namely VNC, SPICE, NBD, migration and character devices.

In this blog series:

Improving QEMU security part 6: TLS support for character devices

Posted: August 16th, 2016 | Author: | Filed under: Coding Tips, Fedora, libvirt, OpenStack, Security, Virt Tools | Tags: , , , | 2 Comments »

This blog is part 6 of a series I am writing about work I’ve completed over the past few releases to improve QEMU security related features.

A number of QEMU device models and objects use a character devices for providing connectivity with the outside world, including the QEMU monitor, serial ports, parallel ports, virtio serial channels, RNG EGD object, CCID smartcard passthrough, IPMI device, USB device redirection and vhost-user. While some of these will only ever need a character device configured with local connectivity, some will certainly need to make use of TCP connections to remote hosts. Historically these connections have always been entirely in clear text, which is unacceptable in the modern hostile network environment where even internal networks cannot be trusted. Clearly the QEMU character device code requires the ability to use TLS for encrypting sensitive data and providing some level of authentication on connections.

The QEMU character device code was mostly using GLib’s  GIOChannel framework for doing I/O but this has a number of unsatisfactory limitations. It can not do vectored I/O, is not easily extensible and does not concern itself at all with initial connection establishment. These are all reasons why the QIOChannel framework was added to QEMU. So the first step in supporting TLS on character devices was to convert the code over to use QIOChannel instead of GIOChannel. With that done, adding in support for TLS was quite straightforward, merely requiring addition of a new configuration property (“tls-creds“) to set the desired TLS credentials.

For example to run a QEMU VM with a serial port listening on IP 10.0.01, port 9000, acting as a TLS server:

$ qemu-system-x86_64 \
      -object tls-creds-x509,id=tls0,endpoint=server,dir=/home/berrange/qemutls \
      -chardev socket,id=s0,host=,port=9000,tls-creds=tls0,server \
      -device isa-serial,chardev=s0
      ...other QEMU options...

It is possible test connectivity to this TLS server using the gnutls-cli tool

$ gnutls-cli --priority=NORMAL -p 9000 \
--x509cafile=/home/berrange/security/qemutls/ca-cert.pem \

In the above example, QEMU was running as a TCP server, and acting as the TLS server endpoint, but this matching is not required. It is valid to configure it to run as a TLS client if desired, though this would be somewhat uncommon.

Of course you can connect 2 QEMU VMs together, both using TLS. Assuming the above QEMU is still running, we can launch a second QEMU connecting to it with

$ qemu-system-x86_64 \
      -object tls-creds-x509,id=tls0,endpoint=client,dir=/home/berrange/qemutls \
      -chardev socket,id=s0,host=,port=9000,tls-creds=tls0 \
      -device isa-serial,chardev=s0
      ...other QEMU options...

Notice, we’ve changed the “endpoint” and removed the “server” option, so this second QEMU runs as a TCP client and acts as the TLS client endpoint.

This feature is available since the QEMU 2.6.0 release a few months ago.

In this blog series:

ANNOUNCE: libosinfo 0.3.1 released

Posted: July 1st, 2016 | Author: | Filed under: Fedora, libvirt, Virt Tools | Tags: , | No Comments »

I am happy to announce a new release of libosinfo, version 0.3.1 is now available, signed with key DAF3 A6FD B26B 6291 2D0E 8E3F BE86 EBB4 1510 4FDF (4096R). All historical releases are available from the project download page.

Changes in this release include:

  • Require glib2 >= 2.36
  • Replace GSimpleAsyncResult usage with GTask
  • Fix VPATH based builds
  • Don’t include autogenerated enum files in dist
  • Fix build with older GCC versions
  • Add/improve/fix data for
    • Debian
    • OpenSUSE
    • FreeBSD
    • Windows
    • RHEL
    • Ubuntu
  • Update README content
  • Fix string comparison for bootable media detection
  • Fix linker flags for OS-X & solaris
  • Fix darwin detection code
  • Fix multiple memory leaks

Thanks to everyone who contributed towards this release.

A special note to downstream vendors/distributors.

The next major release of libosinfo will include a major change in the way libosinfo is released and distributed. The current single release will be replaced with three indepedently released artefacts:

  • libosinfo – this will continue to provide the libosinfo shared library and most associated command line tools
  • osinfo-db – this will contain only the database XML files and RNG schema, no code at all.
  • osinfo-db-tools – a set of command line tools for managing deployment of osinfo-db archives for vendors & users.

The libosinfo and osinfo-db-tools releases will be fairly infrequently as they are today. The osinfo-db releases will be done very frequently, with automated releases made available no more than 1 day after updated DB content is submitted to the project.

ANNOUNCE: virt-viewer 4.0 release

Posted: June 30th, 2016 | Author: | Filed under: Fedora, libvirt, Virt Tools | Tags: , | No Comments »

I am happy to announce a new bugfix release of virt-viewer 4.0 (gpg), including experimental Windows installers for Win x86 MSI (gpg) and Win x64 MSI (gpg). Signatures are created with key DAF3 A6FD B26B 6291 2D0E 8E3F BE86 EBB4 1510 4FDF (4096R)

All historical releases are available from:

Changes in this release include:

  • Drop support for gtk2 builds
  • Require spice-gtk >= 0.31
  • Require glib2 >= 2.38
  • Require gtk3 >= 3.10
  • Require libvirt-glib >= 0.1.8
  • Increase minimum window size fo 320×200 instead of 50×50
  • Remove use of GSLice
  • Don’t show usbredir button if not connected yet
  • Only compute monitor mapping in full screen
  • Don’t ignore usb-filter in spiec vv-file
  • Port to use GtkApplication API
  • Don’t leave window open after connection failure
  • Validate symbols from max glib/gdk versions
  • Don’t use GtkStock
  • Don’t use gtk_widget-modify_{fg,bg} APIs
  • Drop use of built-in eventloop in favour of libvirt-glib
  • Don’t open X display while parsing command line
  • Fix window title
  • Use GResource for building ui files into binary
  • Fix crash with invalid spice monitor mapping
  • Add dialog to show file transfer progress and allow cancelling
  • Remove unused nsis installer support
  • Include adwaita icon theme in msi builds
  • Add more menu mnemonics
  • Fix support for windows consoles to allow I/O redirection
  • Add support for ovirt sso-token in vv-file
  • Fix crash with zooming window while not connected
  • Remove custom auto drawer widget with GtkRevealer
  • Add appdata file for gnome software
  • Misc other bug fixes
  • Refresh translations

Thanks to everyone who contributed towards this release.

ANNOUNCE: libvirt switch to time based rules for updating version numbers

Posted: June 14th, 2016 | Author: | Filed under: Fedora, libvirt, OpenStack | Tags: , | No Comments »

Until today, libvirt has used a 3 digit version number for monthly releases off the git master branch, and a 4 digit version number for maintenance releases off stable branches. Henceforth all releases will use 3 digits, and the next release will be 2.0.0, followed by 2.1.0, 2.2.0, etc, with stable releases incrementing the last digit (2.0.1, 2.0.2, etc) instead of appending yet another digit.

For the longer explanation read on…

We have the following rules about when we increment each digit in the version number

no one has any clue about when we should bump this
bump this when some “significant”[*] features appear
bump this on each new master branch release
bump this for stable branch releases

[*] for a definition of “significant” that either no one knows, or that we invent post-update to justify why we changed the digit.

Now consider the actual requirements libvirt has

  • A number that increments on each release from master branches
  • A number that can be further incremented for stable branch releases without clashing with future master branch releases

The micro + extra digits alone deal with our two actual requirements, so one may ask what is the point of the major + minor digits in the version number ?

In 11 years of libvirt development we’ve only bumped the major digit once, and we didn’t have any real reason why we chose to the bump the major digit, instead of continuing to bump the minor digit. It just felt like we ought to have a 1.0 release after 7+ years. Our decisions about when to bump the minor digit have not been that much less arbitrary. We just look at what features are around and randomly decide if any feel “big enough” to justify a minor digit bump.

Way back in the early days of libvirt, we had exactly this kind of mess when deciding when to actually make releases. Sometimes we’d release after a month, sometimes after 3 months, completely arbitrarily based on whether the accumulated changes felt “big enough” to justify a release. Feature based release schedules are insanity as no one can predict when the next one might happen. Fortunately we wised up pretty quickly and adopted a time base release schedule where we release monthly approximately on the 1st. The only exception is over xmas/new year period, where we avoid Jan 1st and Feb 1st releases and instead have a Jan 15th release, giving a 6 week gap. There is no stated semantic difference between any of our releases off git master branch – they just include whatever happens to be ready at the time.

Considering version numbers again, it is clear that the reason why a feature based release timeline are a bad idea, is just as applicable to feature based version numbering rules. So we have decided to switch to a time based rule for incrementing the version number. Note, that this is *not* to be confused with switching to a time based version number. We want individual digits in the version number to be completely devoid of any semantics. Just as we don’t want version number changes to imply a particular level of feature changes, we also don’t want version numbers to correspond to dates of releases. IOW, we are *not* using the year and month to form the version number, rather that we are using the change in year and change in month as a trigger to update the version number. So our new version number rules are

bumped for the first release of each year
bumped for every major release
bumped for every stable branch release

Rather than wait until January 2017 to put this new rule into effect, we are pretending that July is January, so the next libvirt release will bump the major version number to 2.0.0. There after the releases will be 2.1.0, 2.2.0, etc until January 2017, when we’ll go to 3.0.0.  The maintenance releases based off 2.0.0 will be 2.0.1, 2.0.2, 2.0.3, etc, and live on a v2.0-maint branch in git.

So henceforth you should not interpret the libvirt version numbers as having any semantic meaning. They are merely indicating the progression of releases.

As a reminder, libvirt promises API and ABI stability forever, and the ELF library soname version number is thus fixed forever at, regardless of what version number a release has.