Test-AutoBuild is the oldest open source project of mine that I still actually work on. The original code dates all the way back to approx the year 2000, when Richard Jones and I were working at a now defunct dot-com called BiblioTech (random archive.org historical link). Rich wrote a script called “Rolling Build” which would continuously checkout & build all our software from CVS, publishing the results as RPMs. Thankfully the company allowed the script to be open sourced under the GPLv2+ and I used it as the basis for creating a project called Test-AutoBuild in ~2004. We expanded the code to cover many different SCM tools, maintain historical archives of builds to avoid rebuilding modules if no code had changed and many other things besides. I did a couple of releases a year for while, but it has been on the backburner for the last couple of years. With the increasing number of inter-related virtualization projects using to libvirt & KVM I decided it was time to put Test-AutoBuild back into use as a build server. Yes there are many of automated build systems in existence these days I could have chosen, but I was looking for an excuse to hack on mine again :-)
There was a quiet, mostly unannounced release 1.2.3 back at the start of August, primarily fix the utterly broken GIT support I originally wrote. A couple of weeks ago, immediately before going on holiday, I uploaded release 1.2.4 to the CPAN distribution page. Aside from fixing a number of horrible bugs, the 1.2.4 release brought in the ability to rsync the build results pages to a remote server, so now it is now possible to run the automated builds inside one (or more) private virtual machines and publish the results to a separate public webserver. The second major change was the incorporation of a new theme for the Test-AutoBuild project website and the build status pages. Previously we had used a pretty lame icon of a gear wheel as the logo and a fairly plain web site style. Looking around for some better ideas I happened to come across a proposal for a Fedora 10 Theme that was never taken up. Nicu Buculei and Máirín Duffy, who produced that artwork, were generous enough to grant me permission to use the graphics for Test-AutoBuild under the CC-BY-SA 3.0 and GPLv2+ licenses. Thus for the 1.2.4 release the status pages for the automated builds have been completely restyled and the project’s main website has been similarly updated.
With the 1.2.4 release out and updated RPMs pushed into Fedora, I’m now able to publish the results of the automated builds I run for nearly all the libvirt related virtualization projects. The builder currently runs in a Fedora 14 virtual machine. The plan is to install further virtual machines running important target OS, at the very least, Debian and one of the BSDs, so we can be sure we aren’t causing regressions in our codebases. If I’m feeling adventurous I might even setup a QEMU PPC instance to run some builds on a non-x86 architecture, though that will probably be painfully slow :-)
I’ve written before about how virtualization causes pain wrt keyboard handling and about the huge number of scancode/keycode sets you have to worry about. Following on from that investigative work I completely rewrote GTK-VNC’s keycode handling, so it is able to correctly translate the keycodes it receives from GTK on Linux, Win32 and OS-X, even when running against a remote X11 server on a different platform. In doing so I made sure that the tables used for doing conversions between keycode sets were not just big arrays of magic numbers in the code, as is common practice across the kernel or QEMU codebase. Instead GTK-VNC now has a CSV file containing the unadulterated mapping data along with a simple script to split out mapping tables. This data file and script has already been reused to solve the same keycode mapping problem in SPICE-GTK.
Fast-forward a year and a libvirt developer from Fujitsu is working on a patch to wire up QEMU’s “sendkey” monitor command to a formal libvirt API. The first design question is how should the API accept the list of keys to be injected to the guest. The QEMU monitor command accepts a list of keycode names as strings, or as keycode values as hex-encoded strings. The QEMU keycode values come from what I term the “RFB” codeset, which is just the XT codeset with a slightly unusual encoding of extended keycodes. VirtualBox meanwhile has an API which wants integer keycode values, from the regular XT codeset.
One of the problems with the XT codeset is that no one can ever quite agree on what is the official way to encode extended keycodes, or whether it is even possible to encode certain types of key. There is also a usability problem with having the API require a lowlevel hardware oriented keycode set as input, in that as an application developer you might know what Win32 virtual keycode you want to generate, but have no idea what the corresponding XT keycode is. It would be preferable if you could simply directly inject a Win32 keycode to a Windows guest, or directly inject a Linux keycode to a Linux guest, etc.
After a little bit of discussion we came to the conclusion that the libvirt API should accept an array of integer keycodes, along with a enum parameter specifying what keycode set they belong to. Internally libvirt would then translate from whatever keycode set the application used, to the keycode set required by the hypervisor’s own API. Thus we got an API that looks like:
typedef enum {
VIR_KEYCODE_SET_LINUX = 0,
VIR_KEYCODE_SET_XT = 1,
VIR_KEYCODE_SET_ATSET1 = 2,
VIR_KEYCODE_SET_ATSET2 = 3,
VIR_KEYCODE_SET_ATSET3 = 4,
VIR_KEYCODE_SET_OSX = 5,
VIR_KEYCODE_SET_XT_KBD = 6,
VIR_KEYCODE_SET_USB = 7,
VIR_KEYCODE_SET_WIN32 = 8,
VIR_KEYCODE_SET_RFB = 9,
VIR_KEYCODE_SET_LAST,
} virKeycodeSet;
int virDomainSendKey(virDomainPtr domain,
unsigned int codeset,
unsigned int holdtime,
unsigned int *keycodes,
int nkeycodes,
unsigned int flags);
As with all libvirt APIs, this is also exposed in the virsh command line tool, via a new “send-key” command. As you might expect, this accepts a list of integer keycodes as parameters, along with a keycode set name. If the keycode set is omitted, we are assuming use of the Linux keycode set by default. To be slightly more user friendly though, for the Linux, Win32 & OS-X keycode sets, we also support symbolic keycode names as an alternative to the integer values. These names are simply the name of the #define constant from corresponding header file.
Some examples of how to use the new virsh command are
# send three strokes 'k', 'e', 'y', using xt codeset
virsh send-key dom --codeset xt 37 18 21
# send one stroke 'right-ctrl+C'
virsh send-key dom KEY_RIGHTCTRL KEY_C
# send a tab, held for 1 second
virsh send-key --holdtime 1000 0xf
So when interacting with virtual guests you now have a choice of how to send fake keycodes. If you have a VNC or SPICE connection directly to the guest in question, you can inject keycodes over that channel, while if you have a libvirt connection to the hypervisor you can inject keycodes over that channel.
A couple of years ago Dan Walsh introduced the SELinux sandbox which was a way to confine what resources an application can access using SELinux and the Linux filesystem namespace functionality. Meanwhile we developed sVirt in libvirt to confine QEMU virutal machines, and QEMU itself has gained support for passing host filesystems straight through to the guest operating system, using a VirtIO based transport for the 9p filesystem. This got me thinking about whether it was now practical to create a sandbox based on QEMU, or rather KVM by booting a guest with a root filesystem pointing to the host’s root filesystem (readonly of course), combined with a couple of overlays for /tmp and /home, all protected by sVirt.
One prominent factor in the practicality is how much time the KVM and kernel startup sequences add to the overall execution time of the command being sandboxed. From Richard Jones‘ work on libguestfs I know that it is possible to boot to a functioning application inside KVM in < 5 seconds. The approach I take with 9pfs has a slight advantage over libguestfs because it does not occur the initial (one-time only per kernel version) delay for building a virtual appliance based on the host filesystem, since we’re able to direct access the host filesystem from the guest. The fine details will have to wait for a future blog post, but suffice to say, a stock Fedora kernel can be made to boot to the point of exec()ing the ‘init’ binary in the ramdsisk in ~0.6 seconds, and the custom ‘init’ binary I use for mounting the 9p filesystems takes another ~0.2 seconds, giving a total boot time of 0.8 seconds.
Boot up time, however, is only one side of the story. For some application sandboxing scenarios, the shutdown time might be just as important as startup time. I naively thought that the kernel shutdown time would be unmeasurably short. It turns out I was wrong, big time. Timestamps on the printk messages showed that the shutdown time was in fact longer than the bootup time ! The telling messages were:
[ 1.486287] md: stopping all md devices.
[ 2.492737] ACPI: Preparing to enter system sleep state S5
[ 2.493129] Disabling non-boot CPUs ...
[ 2.493129] Power down.
[ 2.493129] acpi_power_off called
which point a finger towards the MD driver. I was sceptical that the MD driver could be to blame, since my virtual machine does not have any block devices at all, let alone MD devices. To be sure though, I took a look at the MD driver code to see just what it does during kernel shutdown. To my surprise the answer to blindly obvious:
static int md_notify_reboot(struct notifier_block *this, unsigned long code, void *x)
{
struct list_head *tmp;
mddev_t *mddev;
if ((code == SYS_DOWN) || (code == SYS_HALT) || (code == SYS_POWER_OFF)) {
printk(KERN_INFO "md: stopping all md devices.\n");
for_each_mddev(mddev, tmp)
if (mddev_trylock(mddev)) {
/* Force a switch to readonly even array
* appears to still be in use. Hence
* the '100'.
*/
md_set_readonly(mddev, 100);
mddev_unlock(mddev);
}
/*
* certain more exotic SCSI devices are known to be
* volatile wrt too early system reboots. While the
* right place to handle this issue is the given
* driver, we do want to have a safe RAID driver ...
*/
mdelay(1000*1);
}
return NOTIFY_DONE;
}
In other words, regardless of whether you actually have any MD devices, it’ll impose a fixed 1 second delay into your shutdown sequence :-(
With this kernel bug fixed, the total time my KVM sandbox spends running the kernel is reduced by more than 50%, from 1.9s to 0.9s. The biggest delay is now down to Seabios & QEMU which together take 2s to get from the start of QEMU main(), to finally jumping into the kernel entry point.
Traditionally, if you have manually launched a QEMU process, either as root, or as your own user, it will not be visible to/from libvirt’s QEMU driver. This is an intentional design decision because given an arbitrary QEMU process, it is very hard to determine what its current or original configuration is/was. Without knowing QEMU’s configuration, it is hard to reliably perform further operations against the guest. In general, this limitation has not proved a serious burden to users of libvirt, since there are variety of ways to launch new guests directly with libvirt whether graphical (virt-manager) or command line driven (virt-install).
There are always exceptions to the rule, though, and one group of users who have found this a problem is the QEMU/KVM developer community itself. During the course of developing & testing QEMU code, they often have need to quickly launch QEMU processes with a very specific set of command line arguments. This is hard to do with libvirt, since when there is a choice of which command line parameters to use for a feature, libvirt will pick one according to some predetermined rule. As an example, if you want to test something related to the old style ‘-drive’ parameters with QEMU, and libvirt is using the new style ‘-device + -drive’ parameters, you are out of luck & will not be able to force libvirt to use the old syntax. There are other features of libvirt that QEMU developers may well still want to take advantage of though like virt-top or virt-viewer. Thus it is desirable to have a way to launch QEMU with arbitrary command line arguments, but still use libvirt.
A little while ago we did introduce support for adding extra QEMU specific command line arguments in the guest XML configuration, using a separate namespace. This is not entirely sufficient, or satisfactory for the particular scenario outlined above. For this reason, we’ve now introduced a new QEMU-specific API into libvirt that allows the QEMU driver to attach to an externally launched QEMU process. This API is not in the main library, but rather in the separate libvirt-qemu.so library. Use of this library by applications is strongly discouraged and many distros will not supports its use in production deployments. It is intended primarily for developer / troubleshooting scenarios. This QEMU specific command is also exposed in virsh, via a QEMU specific command ‘qemu-attach‘. So now it is possible for the QEMU developers to launch a QEMU process and connect it to libvirt
$ qemu-kvm \
-cdrom ~/demo.iso \
-monitor unix:/tmp/myexternalguest,server,nowait \
-name myexternalguest \
-uuid cece4f9f-dff0-575d-0e8e-01fe380f12ea \
-vnc 127.0.0.1:1 &
$ QEMUPID=$!
$ virsh qemu-attach $QEMUPID
Domain myexternalguest attached to pid 14725
Once attached, most of the normal libvirt commands and tools will be able to at least see the guest. For example, query its status
$ virsh list
Id Name State
----------------------------------
1 myexternalguest running
$ virsh dominfo myexternalguest
Id: 1
Name: myexternalguest
UUID: cece4f9f-dff0-575d-0e8e-01fe380f12ea
OS Type: hvm
State: running
CPU(s): 1
CPU time: 15.1s
Max memory: 65536 kB
Used memory: 65536 kB
Persistent: no
Autostart: disable
Security model: selinux
Security DOI: 0
Security label: unconfined_u:unconfined_r:unconfined_qemu_t:s0-s0:c0.c1023 (permissive)
libvirt reverse engineers an XML configuration for the guest based on the command line arguments it finds for the process in /proc/$PID/cmdline and /proc/$PID/environ. This is using the same code as available via the virsh domxml-from-native command. The important caveat is that the QEMU process being attached to must not have had its configuration modified via the monitor. If that has been done, then the /proc command line will no longer match the current QEMU process state
$ virsh dumpxml myexternalguest
<domain type='kvm' id='1'>
<name>myexternalguest</name>
<uuid>cece4f9f-dff0-575d-0e8e-01fe380f12ea</uuid>
<memory>65536</memory>
<currentMemory>65536</currentMemory>
<vcpu>1</vcpu>
<os>
<type arch='i686' machine='pc-0.14'>hvm</type>
</os>
<features>
<acpi/>
<pae/>
</features>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-kvm</emulator>
<disk type='file' device='cdrom'>
<source file='/home/berrange/demo.iso'/>
<target dev='hdc' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='1' unit='0'/>
</disk>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='5901' autoport='no' listen='127.0.0.1'/>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</memballoon>
</devices>
<seclabel type='static' model='selinux' relabel='yes'>
<label>unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023</label>
</seclabel>
</domain>
Tools like virt-viewer will be able to attach to the guest just fine
$ ./virt-viewer --verbose myexternalguest
Opening connection to libvirt with URI
Guest myexternalguest is running, determining display
Guest myexternalguest has a vnc display
Opening direct TCP connection to display at 127.0.0.1:5901
Finally, you can of course kill the attached process
$ virsh destroy myexternalguest
Domain myexternalguest destroyed
The important caveats when using this feature are
- The guest config must not be modified using monitor commands between the time the QEMU process is started and when it is attached to the libvirt driver
- There must be a monitor socket for the guest using the ‘unix’ protocol as shown in the example above. The socket location does not matter, but it must be a server socket
- It is strongly recommended to specify a name using ‘-name’, otherwise libvirt will auto-assign a name based on the $PID
To re-inforce the earlier point, this feature is ONLY targetted at QEMU developers and other people who want to do ad-hoc testing/troubleshooting of QEMU with precise control over all command line arguments. If things break when using this feature, you get to keep both pieces. To re-inforce this, when attaching to an externally launched guest, it will be marked as tainted which may limit the level of support a distro / vendor provides. Anyone writing serious production quality virtualization applications should NEVER use this feature. It may make babies cry and kick cute kittens.
This feature will be available in the next release of libvirt, currently planned to be version 0.9.4, sometime near to July 31st, 2011
This blog post isn’t exactly “news” to many readers, but I feel it is worth reminding people of the risks of letting their browser remember form data history
Being reasonably paranoid about security / privacy, one of the first things I do when starting firefox on a freshly installed machine is to go to the preferences and change the history settings so that it does not ever remember any form data. Unfortunately on my most recently installed laptop I had forgotten to do this, for inexplicable reasons, so firefox was happily remembering form data. Before clearing the data out, I decided to take a look at just what firefox had remembered over the past couple of months.
- Card numbers: Every credit and debit card number
- CVV/CVC: The 3 digit verification codes from the back of several cards
- Social security / national insurance numbers
- Bank accounts: Several bank account numbers and sort codes
- Addresses: Home address, amongst others
- Date of birth: Handy in combination with the above data
- User names: login name for many many websites, even those where I said not to remember the actual passwords
- Phone numbers: my home and mobile phone numbers, amongst others
- Answers to several “security questions” used to reset passwords on sites
You might wonder where/how firefox keeps all this sensitive data ? In a sqlite database with no encryption in your firefox profile ($HOME/.mozilla/firefox/XXXXXXXX.default/formhistory.sqlite). You can look at what is stored using the sqlite3 command line tool:
# sqlite3 formhistory.sqlite
sqlite> select fieldname, value from moz_formhistory order by value;
Or, if you trust the plugin author, you can install the firefox “Form History Control” addon which lets you browse all the data from a nice UI.
While I do have my laptop disk running with dm-crypt block device encryption, I still have a golden rule that no sensitive data is stored in cleartext in files since, while the OS is running, disk encryption is no real barrier malicious programs that find their way on to the machine. If it is a work provided machine, the data may also be finding its way into the corporate backup system & who knows if that is encrypted well enough. Any files with sensitive data have additional GPG encryption and really sensitive data will be on a separate encrypted USB stick, again with extra encryption for the files themselves. Keeping form history in an unencrypted sqlite database in my laptop home directory clearly violates my security goals. Why would a $EVIL criminal go the trouble of creating a virus which installs a keystroke logger in someone’s browser, when they can get a veritable goldmine of data by simply taking a copy of the formhistory.sqlite file.
Having known about this problem since 2003, a “fix” was applied in 2010 for firefox 4 which attempts to identify credit card numbers in forms and not store them in the form history. Great, now what about all the other data it is storing some of which is just as sensitive as credit card numbers, if not more so ? If credit card details get mis-used, liability is usually on the credit card company, but not so for social security numbers, bank account numbers, etc…
Needless to say, I have fixed my preferences in firefox so that it doesn’t remember form data, and deleted the formhistory.sqlite file to remove any existing stored data it had. I have to wonder how many users of firefox are out there running with the default history settings, without any realization that a great deal of their sensitive personal data is being silently stored in plain sight for any malicious virus/trojan to capture. The annoying thing is form history can be really useful for certain websites where there’s alot of mundane non-sensitive data that needs entering frequently. As the last commenter on the bug 188285 proposed, it would be desirable if firefox prompted before remembering form data with options “Yes, Not at this time, Never for this site”. It would also be desirable for data in the formhistory.sqlite file to be encrypted as firefox can already do when remembering passwords.
To stop firefox recording your form history:
- Select menu Edit -> Preferences
- Navigate to tab Privacy
- Change Firefox will: Remember history to Firefox will: Use custom settings for history
- Unselect Remember search and form history
To clear your existing form history
- Select menu Tools -> Clear Recent History
- Change Time range to clear: Last Hour to Time range to clear: Everything
- Expand details
- Make sure Form History is selected (optionally unselect other bits you don’t want cleared)
- Press Clear Now
If you are super paranoid, then shutdown firefox and delete the file $HOME/.mozilla/firefox/XXXXXXXX.default/formhistory.sqlite and then start firefox again