Since the project’s creation about 14 years ago, libvirt has grown enormously. In that time there has been a lot of code refactoring, but these were always fairly evolutionary changes; there has been little revolutionary change of the overall system architecture or some core technical decisions made early on. This blog post is one of a series examining recent technical decisions that can be considered more revolutionary to libvirt. This was the topic of a talk given at KVM Forum 2019 in Lyon.
Detecting and reporting OOM
Libvirt has always taken the view that ANY error from a function / system call must be propagated back to the caller. The out of memory condition (ENOMEM / OOM) is just one of many errors that might be seen when calling APIs, and thus libvirt attempted to report this in the normal manner. OOM is not like most other errors though.
The first challenge with OOM is that checking for a NULL return from malloc() is error prone because the return value overloads the error indicator with the normal returned pointer. To address this libvirt coding style banned direct use of malloc() and created a wrapper API that returned the allocated pointer in an output parameter, leaving the return value solely as the error indicator leading to a code pattern like:
char *varname;
if (VIR_ALLOC(varname) < 0) {
....handle OOM...
}
This enabled use of the ‘return_check‘ function attribute to get compile time validation that allocation errors were checked. Checking for OOM is only half the problem. Handling OOM is the much more difficult issue. Libvirt uses a ‘goto error‘ design pattern for error cleanup code paths. A surprisingly large number of these goto jumps only exist to handle OOM cleanup. Testing these code paths is non-trivial, if not impossible, in the real world. Libvirt integrated a way to force OOM on arbitrary allocations in its unit test suite. This was very successful at finding crashes and memory leaks in OOM handling code paths, but this only validates code that actually has unit test coverage. The number of bugs found in code that was tested for OOM, gives very low confidence that other non-tested code would correctly handle OOM. The OOM testing is also incredibly slow to execute since it needs to repeatedly re-run the unit tests failing a different malloc() each time. The time required grows exponentially as the number of allocations increases.
Assuming the OOM condition is detected and a jump to the error handling path is taken, there is now the problem of getting the error report back to the user. Many of the libvirt drivers run inside the libvirtd daemon, with an RPC system used to send results back to the client application. Reporting the error via RPC messages is quite likely to need memory allocation which may well fail in an OOM scenario.
Is OOM reporting useful?
The paragraphs above describe why reporting OOM scenarios is impractical, verging on impossible, in the real world. Assuming it was possible to report reliably though, would it actually benefit any application using libvirt ?
Linux systems generally default to having memory overcommit enabled, and when they run out of memory, the OOM killer will reap some unlucky process. IOW, on Linux, it is very rare for an application to ever see OOM reported from an allocation attempt. Libvirt is ported to non-Linux platforms which may manage memory differently and thus genuinely report OOM from malloc() calls. Those non-Linux users will be taking code paths that are never tested by the majority of libvirt users or developers. This gives low confidence for success.
Although libvirt provides a C library API as its core deliverable, few applications are written in C, most consume libvirt via a language binding with Perl and Go believed to be the most commonly used. Handling OOM in non-C languages is even less practical/common than in C. Many libvirt applications are also already using libraries (GTK, GLib) that will abort on OOM. Overall there is little sign that any libvirt client application attempts to handle OOM in its own code, let alone care if libvirt can report it.
One important application process using the libvirt API though is the libvirtd daemon. In the very early days, if libvirtd stopped it would take down all running QEMU VMs, but this limitation was fixed over 10 years ago. To enable software upgrades on hosts with running VMs, libvirtd needs to be able to restart itself. As a result libvirtd maintains a record of important state on disk enabling it to carry on where it left off when starting up. Recovering from OOM by aborting and allowing the libvirtd to be restarted by systemd, would align with a code path that already needs to be well tested and supported for software upgrades.
Give up on OOM handling
With all the above in mind, the decision shouldn’t be a surprise. Libvirt has decided to stop attempting to handle ENOMEM from malloc() and related APIs and will instead immediately abort. The libvirtd daemon will automatically restart and carry on where it left off. The result is that the libvirt code can be dramatically simplified by removing many goto jump and cleanup code blocks, which reduces the maint burden on libvirt contributors, allowing more time to be spent on coding features which matter to users.
I am pleased to announce that a new release of the libvirt-glib package, version 3.0.0, is now available from
https://libvirt.org/sources/glib/
The packages are GPG signed with
Key fingerprint: DAF3 A6FD B26B 6291 2D0E 8E3F BE86 EBB4 1510 4FDF (4096R)
Changes in this release:
- Add support for bochs video device
- Add API to query firmware config
- Improve testing coverage
- Validate min/max glib API versions in use
- Remove deprecated G_PARAM_PRIVATE
- Fix docs build linking problems
- Convert python demos to be python 3 compatible & use modern best practice for pyobject introspection bindings
- Add API to query domain capaibilities
- Refresh translations
- Simplify build process for handling translations
- Fix some memory leaks
- Add API for setting storage volume features
Thanks to everyone who contributed to this new release through testing, patches, bug reports, translations and more.
I’m pleased to announce a new release of GTK-VNC, version 1.0.0.
https://download.gnome.org/sources/gtk-vnc/1.0/gtk-vnc-1.0.0.tar.xz (211K)
sha256sum: a81a1f1a79ad4618027628ffac27d3391524c063d9411c7a36a5ec3380e6c080
Pay particular attention to the first two major changes in
this release:
- Autotools build system replaced with meson
- Support for GTK-2 is dropped. GTK-3 is mandatory
- Support for libview is dropped in example program
- Improvements to example demos
- Use MAP_ANON if MAP_ANONYMOUS doesn’t exist to help certain macOS versions
- Fix crash when connection attempt fails early
- Initialize gcrypt early in auth process
- Emit vnc-auth-failure signal when SASL auth fals
- Emit vnc-error signal when authentication fails
- Fix double free when failing to read certificates
- Run unit tests in RPM build
- Modernize RPM spec
- Fix race condition in unit tests
- Fix install of missing header for cairo framebuffer
- Fix typo in gir package name
- Add missing VncPixelFormat file to gir data
Thanks to all those who reported bugs and provides patches that went into this new release.
QEMU is able to leverage a number of live tracing systems, the choice configurable at build time between
- log – printf formatted string for each event sent into QEMU’s logging system which writes to stderr
- syslog – printf formatted string for each event sent via syslog
- simple – binary data stream for each event written to a file or fifo pipe
- ftrace – printf formatted string for each event sent to kernel ftrace facility
- dtrace – user space probe markers dynamically enabled via dtrace or systemtap
- ust – user space probe markers dynamically enabled via LTT-ng
Upstream QEMU enables the “log” trace backend by default since it is cross-platform portable and very simple to use by adding “-d trace:PATTERN” on the QEMU command line. For example to enable logging of all trace events in the QEMU I/O subsystem (aka “qio“) we can
$ qemu -d trace:qio* ...some args...
23266@1547735759.137292:qio_channel_socket_new Socket new ioc=0x563a8a39d400
23266@1547735759.137305:qio_task_new Task new task=0x563a891d0570 source=0x563a8a39d400 func=0x563a86f1e6c0 opaque=0x563a89078000
23266@1547735759.137326:qio_task_thread_start Task thread start task=0x563a891d0570 worker=0x563a86f1ce50 opaque=0x563a891d9d90
23273@1547735759.137491:qio_task_thread_run Task thread run task=0x563a891d0570
23273@1547735759.137503:qio_channel_socket_connect_sync Socket connect sync ioc=0x563a8a39d400 addr=0x563a891d9d90
23273@1547735759.138108:qio_channel_socket_connect_fail Socket connect fail ioc=0x563a8a39d400
This is very simple and surprisingly effective much of the time, but it is not without its downsides
- Inactive probes have non-negligible performance impact on hot codepaths
- It is targetted to human consumption, so it is not easy to reliably process with machines
- It requires adding arguments to QEMU’s command line so is not easy to enable in many cases
- It is custom to QEMU so does not facilitate getting correlated traces across the whole system
For these reasons, some downstreams chose not to use the default “log” backend. Both Fedora and RHEL have instead enabled the “dtrace” backend which can be leveraged via systemtap on Linux. This provides a very powerful tracing system, but the cost is that the previous simple task of printing a formatted string when a probe point fires has become MUCH more complicated. For example to get equivalent output to that seen with QEMU’s log backend would require
# cat > trace.stp <<EOF
probe qemu.system.x86_64.qio_task_new {
printf("%d@%d qio_task_new Task new task=%p source=%p func=%p opaque=%p\n",
pid(), gettimeofday_ns(), task, source, func, opaque)
}
EOF
# stap trace.stp
22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400
Repeat that code snippet for every qio* probe point you want to watch, figuring out the set of args it has available to print.This quickly becomes tedious for what should be a simple logging job, especially if you need to reference null terminated strings from userspace.
After cursing this difficulty one time too many, it occurred to me that QEMU could easily do more to make life easier for systemtap users. The QEMU build system is already auto-generating all the trace backend specific code from a generic description of probes in the QEMU source tree. It has a format string which is used in the syslog, log and ftrace backends, but this is ignored for the dtrace backend. It did not take much to change the code generator so that it can use this format string to generate a convenient systemtap tapset representing the above manually written probe:
probe qemu.system.x86_64.log.qio_task_new = qemu.system.x86_64.qio_task_new ?
{
printf("%d@%d qio_task_new Task new task=%p source=%p func=%p opaque=%p\n",
pid(), gettimeofday_ns(), task, source, func, opaque)
}
This can be trivially executed with minimal knowledge of systemtap tapset language required
# stap -e "qemu.system.x86_64.log.qio_task_new{}"
22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400
Even better, we have now gained the ability to use wildcards too
# stap -e "qemu.system.x86_64.log.qio*{}"
23266@1547735759.137292:qio_channel_socket_new Socket new ioc=0x563a8a39d400
23266@1547735759.137305:qio_task_new Task new task=0x563a891d0570 source=0x563a8a39d400 func=0x563a86f1e6c0 opaque=0x563a89078000
23266@1547735759.137326:qio_task_thread_start Task thread start task=0x563a891d0570 worker=0x563a86f1ce50 opaque=0x563a891d9d90
23273@1547735759.137491:qio_task_thread_run Task thread run task=0x563a891d0570
23273@1547735759.137503:qio_channel_socket_connect_sync Socket connect sync ioc=0x563a8a39d400 addr=0x563a891d9d90
23273@1547735759.138108:qio_channel_socket_connect_fail Socket connect fail ioc=0x563a8a39d400
Users still, however, need to be aware of the naming convention for QEMU’s systemtap tapsets and how it maps to the particular QEMU binary that is used, and don’t forget the trailing “{}”. Thus I decided to go one step further and ship a small helper tool to make it even easier to use
$ qemu-trace-stap run qemu-system-x86_64 'qio*'
22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00
22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400
22806@1547735341399865943 qio_task_thread_start Task thread start task=0x56135cd66eb0 worker=0x56135af72e50 opaque=0x56135c071d70
22806@1547735341399976816 qio_task_thread_run Task thread run task=0x56135cd66eb0
The second argument to this tool is the QEMU binary filename to be traced, which can be relative (to search $PATH) or absolute. What is clever is that it will set the SYSTEMTAP_TAPSET env variable to point to the right location to find the corresponding tapset definition. This is very useful when you have multiple copies of QEMU on the system and need to make sure systemtap traces the right one.
The ‘qemu-trace-stap‘ script takes a verbose arg so you can understand what it is running behind the scenes:
$ qemu-trace-stap run /home/berrange/usr/qemu-git/bin/qemu-system-x86_64 'qio*'
Using tapset dir '/home/berrange/usr/qemu-git/share/systemtap/tapset' for binary '/home/berrange/usr/qemu-git/bin/qemu-system-x86_64'
Compiling script 'probe qemu.system.x86_64.log.qio* {}'
Running script, <Ctrl>-c to quit
...trace output...
It can enable multiple probes at once
$ qemu-trace-stap run qemu-system-x86_64 'qio*' 'qcrypto*' 'buffer*'
By default it monitors all existing running processes and all future launched proceses. This can be restricted to a specific PID using the –pid arg
$ qemu-trace-stap run --pid 2532 qemu-system-x86_64 'qio*'
Finally if you can’t remember what probes are valid it can tell you
$ qemu-trace-stap list qemu-system-x86_64
ahci_check_irq
ahci_cmd_done
ahci_dma_prepare_buf
ahci_dma_prepare_buf_fail
ahci_dma_rw_buf
ahci_irq_lower
...snip...
This new functionality merged into QEMU upstream a short while ago and will be included in the QEMU 4.0 release coming at the end of April.
I am pleased to announce a new release 2.0 of Entangle is available for download from the usual location:
https://entangle-photo.org/download/
This release is largely bug fixes with a couple of small features
- Require gobject introspection >= 1.54
- Require GTK3 >= 3.22
- Fix dependency on libraw
- Fix variable name in photobox plugin
- Document some missing keyboard shortcuts
- Fix upper bound in histogram to display clipped pixel
- Refresh translations
- Option to highlight over exposed pixels in red
- Disable noisy compiler warning
- Remove use of deprecated application menu concept
- Fix image redraw when changing some settings
- Update mailing list address in appdaat
- Add more fields to appdata content
- Fix refrence counting during window close
- Use correct API for destroying top level windows
- Fix unmounting of cameras with newer gvfs URI naming scheme
- Avoid out of bounds read of property values
- Fix many memory leaks
- Workaround for combo boxes not displaying on Wayland
- Fix race condition in building enums
- Fix setting of gschema directory during startup
- Set env to ensure plugins can find introspection typelib Requires