One of the most frequently requested features for both libvirt and virt-manager is the ability to manage virtual machines remotely. When considering this problem, one immediately thinks of the security implications – most of the communications being used for local management have minimal security. For example, with Xen the XenD offers access over TCP with either XML-RPC or a custom SEXPR protocol. In both cases though, the communication channel is unencrypted and the authentication is minimal. Even though the new XML-RPC protocol does offer authentication against PAM this is totally unsuitable for use off-host because the passwords would be transmitted in clear text :-( The core management APIs are not the only area using a cleartext communication channel – the migration of virtual machines between hosts is also done in the clear – not great if the OS running in the VM has any sensitive data in memory. The virtual console capability is based on the VNC remote framebuffer protocol – again in everything is being transmitted in cleartext.
When raising the question of secure remote management, a frequently suggested solution is to just tunnel everything over SSH. While this would obviously address the issue of wire level encryption just fine, it is less desirable when considering the issues of authentication, access control, and key management. To manage virtual machines using SSH tunnelling one would have to create user accounts on each Dom0 for anyone who can manage VMs. Integrity / security of the Dom0 hosts is critical in a virtualized environment because any single host is running many VMs, so giving people shell access with SSH is not at all desirable. It is possible to restrict a users’ access by only allowing authentication with public keys, and using magic in the authorized_keys file to whitelist a set of commands that can be run.
That said, in a data center with 1000’s hosts, one certainly does not want to be maintaining 10’s of 1000’s of authorized_keys files – just think of the work involved in revoking access to an admin who resigns / is fired. Authentication via SSH is at a rather coarse level – the user can either log into the host, or not – there’s no facility for saying ‘user A can manage virtual machines B and C, but not virtual machine D or E’. So there would need to be a second level of access control on top of that provided by SSH. Thus as well as managing 10’s of 1000’s of authorized_keys files, there’s the additional management hassle of users in the management protocol itself. While on the subject of SSH, when logging into a machine for the first time, how many people actually verify that the host key is correct ? Not many I suspect. I know I usually just type ‘yes’ every time.
So IMHO, using SSH as the foundation for secure remote management of virtual machines in a large data center is not really practical. The answer to these problems is to utilize TLS (the successor to SSL). Contrary to popular belief TLS is not just used for web servers/browsers / HTTPS – it is a general purpose protocol which can be layered onto pretty many any network communication protocol with surprisingly little effort. So, how does it deal with the issues identified above ? WRT to the question of host key checking, the key is that TLS uses X509 certificates and with that comes the concept of a Certificate Authority. Apriori, one only has to trust the certificate authority (CA). Thus, when connecting to a server for the first time, one validates the certificate presented by checking that it was signed by the trusted CA, and compare the server hostname against the ‘common name’ embedded in the signed certificate. Obviously if one is communicating directly using the virtualization management server’s native protocol there is no need to give any shell access to the Dom0 host. Each client connecting to the server would have their own ‘client certificate’ which is signed by the certificate authority. Upon accepting a client connection, the server validates the CA signature on the client’s certificate, and also checks the ‘certificate revokation list’ (CRL) published by the CA. Assuming the CRL is pushed to each Dom0 host on a periodic basis, there is only one place the admin needs to go to revoke a user’s access to all hosts – the CA admin site. Fine grained access control per VM, can be keyed off a whitelist of client certificates, either based on the ‘common name’ or ‘fingerprint’ fields in the cert.
Thus my long term vision for remote management is to TLS enable all the network protocols involved. As a first step Rich Jones is working on a TLS enabled daemon for libvirt which will allow apps to securely manage any virtualization driver supported by libvirt.
This is the core first step in getting virt-manager operating remotely. The obvious next step in this process is to enable TLS in the VNC protocol. The main open source VNC server, RealVNC, does not offer any standard TLS support – they restrict this stuff to the commercial version. Over the years several people have offered patches to add TLS to VNC, but they’ve never been accepted in the RealVNC codebase, which has unfortunately led to a fork – VeNCrypt. Thus for QEMU and the VNC daemon used by Xen paravirt framebuffer, I’d anticipate implementing the VNC protocol extensions defined by VeNCrypt to add TLS support. I’m currently hacking on a GTK VNC viewer widget to replace the current VNC viewer in virt-manager, with the express purpose of supporting TLS. The final stage of the plan will involve adapting the Xen and QEMU migration protocols to be layered over a TLS connection. There are many interesting questions / challenges to be worked out along the way, and certainly a hell of alot of coding to be done. The payoff at the end will be well worth it though.
As I mentioned earlier, Rich Jones is working on the libvirt management protocol, and I’m attacking the VNC client side of things. That still leaves a lot of hacking to be done – the server side VNC TLS impl in QEMU and the Xen framebuffer daemon are both sizeable chunks of work, and I’m not aware of anyone tackling the migration stuff either. So there’s plenty of coding work for people who are interested in this….
Sam writes
Per a limitation in Xen, Xen guests have no support for USB at all. Many people wondered how guest domains would handle a USB thumbdrive. Short answer: they wont.
This isn’t the entire truth. For paravirtualized guests it is possible to do device hot plug with virtual disks. So a USB thumbdrive plugged into the host could be mapped into the guest as a virtual disk. If run locally on Dom0, I could imagine virt-manager listening in to DBus events from HAL about newly inserted USB storage devices. The user could be prompted to map the device through to a particular guest – if we remember the device’s UUID (universal unique identifier) we could automatically map it through to the same guest on subsequent insertion. The tricky aspect to all of this though is of course the removal process. The user would have to remember to umount the device in the guest before virt-manager in Dom0 unmapped it from the guest, finally enabling the user to physically remove it. Oh, we’d also have to figure out how to block GNOME volume manager from automatically mounting it in the host. So while there’s no direct USB support, it could be fun to hack up some form of integration between HAL, virt-manaager and Xen paravirt guests to deal with USB storage hotplug.
On the other hand if someone did want to write a virtual USB split front/back device driver for Xen, that’d be excellant too. Although far from trivial…
After creating the aforementioned $900 paperweight, I then spent another few hours turning it back into a useful computer again. If anyone should find themselves wanting todo a ‘from scratch’ dual boot install of Mac OS-X and Fedora, it is actually quite simple….
The first step was to get Mac OS-X back onto the system. So I inserted the original install disk, and held down ‘c’ while turning on the mini. A short while later the Mac OS-X installer is up & running. Before beginning the install process, I launched the ‘Disk Utility’ GUI tool. Using its partitioning options I deleted all existing partitions then told it to create 1 partition of 36 GB marked for a HFS+ filesystem, and leave the remaining 30-something GB as unallocated free space. Quitting out of Disk Utility, I now started with the standard Mac OS-X install wizard, letting it install to this 36 GB partition. This all went very smoothly and 30 minutes later I had fully functional Mac OS-X installed & booting correctly.
Next step is to install Fedora, but before doing this I decided to setup the rEFIt bootloader. The automated setup process for rEFIt installs it onto the main Mac OS-X HFS+ filesystem. I’m not sure whether I’ll keep Mac OS-X installed long term and don’t fancy loosing the bootloader if I re-install that partition. This is a perfect solution to this – install rEFIt onto the hidden 200 MB EFI system partition. The rEFIt install instructions (unhelpfully) decline to tell you how to achieve this, but thanks to Google I came across a guy who has documented the process. Went through that (very simple) process, rebooted and bingo – rEFIt is there, showing a single boot option – for Mac OS-X. So now it is time to try installing Fedora again
Inserting the Fedora CD and rebooting while holding down ‘c’ starts up the Fedora installer. Since I had taken care to leave a chunk of unpartitioned free space earlier, this I could simply let anaconda do its default partitioning – which meant no need to play with lvm tools manually. I had wanted to setup a separate /home, but since anaconda was in text mode there’s no way to add/remove LVM volumes. Still it was possible to shrink down the root filesystem size, leaving the majority of the LVM volume group unallocated for later use. Once past the partitioning step, the remainder of the installer was straightforward – just remember to install grub in the first sector of /boot, and not in the MBR. 30 minutes later I rebooted and to my delight rEFIt showed options for booting both Mac OS-X and Fedora. Success!
Once Fedora was up and running there was only one remaining oddity to deal with. The Mac Mini has a i945 Intel graphics card in it, which isn’t supported by the i810 driver that is into the current released of Xorg. Fortunately Fedora ships a pre-release of the ‘intel’ driver which does support the i945 and does automatic mode setting. So it ought to ‘just work’, but it didn’t. I should mention at this point, that the Mac Mini is not connected to a regular monitor, its actually going to my Samsung LCD HDTV, which has a native resolution of 1360×768. After poking around in the Xorg logs, I discovered that the TV wasn’t returning any EDID info, so the graphics driver didn’t have any info with which to generate suitable modelines. The manual for the TV says it requires a standard VESA 1360×768 resolution. A little googling later I found a suitable modeline, added it to the xorg.conf and X finally starts up just fine at the native resolution. For anyone else out there with a Samsung LN-S3241D widescreen HDTV, the xorg.conf sections that did the trick look like this:
Section "Monitor"
Identifier "TV0"
HorizSync 30-60
VertRefresh 60-75
ModeLine "1360x768@60" 85.800 1360 1424 1536 1792 768 771 777 795 +HSync +VSync
EndSection
Section "Screen"
Identifier "Screen0"
Device "Videocard0"
Monitor "TV0"
DefaultDepth 24
SubSection "Display"
Viewport 0 0
Modes "1360x768@60"
Depth 24
EndSubSection
EndSection
So compared to the pain involved in breaking in the Mac Mini, bringing it back to life was a quite uneventful affair. And if I had done the install while connected to a regular monitor instead of TV, it would have been even simpler. Anyway, I’m very happy to have both Mac OS-X & Fedora Core 6 runnning – the latter even has the desktop effects bling, and Xen with fully-virt working.
This weekend I decided it was way overdue to switch my Intel Mac Mini over to using Fedora. I’d read Fedora on Mactel notes and although they’re as clear as mud, it didn’t sound like it should be much trouble.
First off I applied all the available Mac OS-X updates, installed Bootcamp and resized the HFS+ partition to allow 36 GB of space for Linux. Bootcamp unfortunately isn’t too smart and so assumed I wanted to install Windows. No matter, once I’d resized the partition I could quit out of the Bootcamp wizard and take care of the details myself. So I stuck the Fedora Core 6 install CD, and used the GUI to change the startup disk to be the CD – this was the first stupid thing I did – I should have simply held down ‘c’ at the next boot instead of changing the default startup disk.
Anyway, so it booted into the installer, but Anaconda failed to start an X server, so it took me into text mode installation process. I figured this was because of the funky i810 graphics card so didn’t worry really. Until I came to the partitioning stage – I had a single partition available /dev/sda3 but BootCamp had marked this as a Windows partition – so neither the ‘Remove all Linux partitions’ or ‘Use unallocated space’ options would do the trick. And because this was text mode, I couldn’t manually paritition because none of LVM UI is available. No problem, I’ll just switch into the shell and use ‘fdisk’ and ‘lvm’ to setup partitioning in the old school way. I know now, this was a huge mistake :-) Its not explicitly mentioned in the Fedora on Mactel, but Mac OS-X uses the GPT partition table format, not the tradition DOS MBR style. It does provide an emulated MBR so legacy OS’ can see the partitions, but this should be considered read-only. Unfortunately I wasn’t paying attention, so happily run fdisk, deleting the Windows partition created by bootcamp, and creating several new ones to use for /boot and the LVM physical volume. There was also a wierd 200 MB vfat partition at the start of the disk which I hadn’t asked for ever, so I decided to repurpose that for /boot.
I then tried to setup LVM, but it kept complaining the device /dev/sda4 didn’t exist – but fdisk clearly said it did exist. Perhaps the kernel hadn’t re-read the partition table, so I played with fdisk a bit more to no avail, and then rebooted and re-entered the installer again. The kernel still only saw the original 3 partitions, but oddly fdisk did see the extra 4th partition.
I google’d around and found that the Mac OS-X install disk had a GUI partitioning tool, so decided I’d use that to delete the non-HFS paritions, just leaving free unallocated space, which would let Anaconda do automagic partitioning. This seemed to work – I did manage to get through the complete Fedora instal – but after rebooting I discovered something horribly wrong. The BIOS was still configured to boot off CD image. Opps. No matter, I held down the magic key to invoke Bootcamp, but Bootcamp not only wouldn’t see the Fedora install I just did, but also wouldn’t see my Mac OS-X install. All it offered was the choice to boot Windows – which I had never installed ! Of course that failed.
At this point I burnt a bootable CD with rEFIt on it success I could now see my Fedora install and boot it, but still no sign of Mac OS-X :-( Also I didn’t really want to have to leave a rEFIt CD in the drive for every boot. This is when I discovered that the 200 MB vfat partition I repurposed for /boot was in fact used by the EFI BIOS, and things like rEFIt. Doh. I could reformat it as vfat manually, but to install rEFIt into it, requires some wierd operation called ‘blessing’ which was only documented using Mac OS-X tools.
I figured my best option now was to boot the Mac OS-X install CD and play with the partitioning tool again – maybe this would be able to repair my Mac OS-X partition such that I could boot it again. No such luck. All I managed to do was break the Fedora install I had just completed. The transformation from functioning Mac Mini with Mac OS-X, into $900 paperweight was now complete. I had two OS’s installed, both broken to the extent that neither BootCamp or rEFIt would boot them, and BootCamp offering the option of booting a Windows install which didn’t exist :-)