QEMU is able to leverage a number of live tracing systems, the choice configurable at build time between
- log – printf formatted string for each event sent into QEMU’s logging system which writes to stderr
- syslog – printf formatted string for each event sent via syslog
- simple – binary data stream for each event written to a file or fifo pipe
- ftrace – printf formatted string for each event sent to kernel ftrace facility
- dtrace – user space probe markers dynamically enabled via dtrace or systemtap
- ust – user space probe markers dynamically enabled via LTT-ng
Upstream QEMU enables the “log” trace backend by default since it is cross-platform portable and very simple to use by adding “-d trace:PATTERN” on the QEMU command line. For example to enable logging of all trace events in the QEMU I/O subsystem (aka “qio“) we can
$ qemu -d trace:qio* ...some args...
23266@1547735759.137292:qio_channel_socket_new Socket new ioc=0x563a8a39d400
23266@1547735759.137305:qio_task_new Task new task=0x563a891d0570 source=0x563a8a39d400 func=0x563a86f1e6c0 opaque=0x563a89078000
23266@1547735759.137326:qio_task_thread_start Task thread start task=0x563a891d0570 worker=0x563a86f1ce50 opaque=0x563a891d9d90
23273@1547735759.137491:qio_task_thread_run Task thread run task=0x563a891d0570
23273@1547735759.137503:qio_channel_socket_connect_sync Socket connect sync ioc=0x563a8a39d400 addr=0x563a891d9d90
23273@1547735759.138108:qio_channel_socket_connect_fail Socket connect fail ioc=0x563a8a39d400
This is very simple and surprisingly effective much of the time, but it is not without its downsides
- Inactive probes have non-negligible performance impact on hot codepaths
- It is targetted to human consumption, so it is not easy to reliably process with machines
- It requires adding arguments to QEMU’s command line so is not easy to enable in many cases
- It is custom to QEMU so does not facilitate getting correlated traces across the whole system
For these reasons, some downstreams chose not to use the default “log” backend. Both Fedora and RHEL have instead enabled the “dtrace” backend which can be leveraged via systemtap on Linux. This provides a very powerful tracing system, but the cost is that the previous simple task of printing a formatted string when a probe point fires has become MUCH more complicated. For example to get equivalent output to that seen with QEMU’s log backend would require
# cat > trace.stp <<EOF
probe qemu.system.x86_64.qio_task_new {
printf("%d@%d qio_task_new Task new task=%p source=%p func=%p opaque=%p\n",
pid(), gettimeofday_ns(), task, source, func, opaque)
}
EOF
# stap trace.stp
22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400
Repeat that code snippet for every qio* probe point you want to watch, figuring out the set of args it has available to print.This quickly becomes tedious for what should be a simple logging job, especially if you need to reference null terminated strings from userspace.
After cursing this difficulty one time too many, it occurred to me that QEMU could easily do more to make life easier for systemtap users. The QEMU build system is already auto-generating all the trace backend specific code from a generic description of probes in the QEMU source tree. It has a format string which is used in the syslog, log and ftrace backends, but this is ignored for the dtrace backend. It did not take much to change the code generator so that it can use this format string to generate a convenient systemtap tapset representing the above manually written probe:
probe qemu.system.x86_64.log.qio_task_new = qemu.system.x86_64.qio_task_new ?
{
printf("%d@%d qio_task_new Task new task=%p source=%p func=%p opaque=%p\n",
pid(), gettimeofday_ns(), task, source, func, opaque)
}
This can be trivially executed with minimal knowledge of systemtap tapset language required
# stap -e "qemu.system.x86_64.log.qio_task_new{}"
22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400
Even better, we have now gained the ability to use wildcards too
# stap -e "qemu.system.x86_64.log.qio*{}"
23266@1547735759.137292:qio_channel_socket_new Socket new ioc=0x563a8a39d400
23266@1547735759.137305:qio_task_new Task new task=0x563a891d0570 source=0x563a8a39d400 func=0x563a86f1e6c0 opaque=0x563a89078000
23266@1547735759.137326:qio_task_thread_start Task thread start task=0x563a891d0570 worker=0x563a86f1ce50 opaque=0x563a891d9d90
23273@1547735759.137491:qio_task_thread_run Task thread run task=0x563a891d0570
23273@1547735759.137503:qio_channel_socket_connect_sync Socket connect sync ioc=0x563a8a39d400 addr=0x563a891d9d90
23273@1547735759.138108:qio_channel_socket_connect_fail Socket connect fail ioc=0x563a8a39d400
Users still, however, need to be aware of the naming convention for QEMU’s systemtap tapsets and how it maps to the particular QEMU binary that is used, and don’t forget the trailing “{}”. Thus I decided to go one step further and ship a small helper tool to make it even easier to use
$ qemu-trace-stap run qemu-system-x86_64 'qio*'
22806@1547735341399856820 qio_channel_socket_new Socket new ioc=0x56135d1d7c00
22806@1547735341399862570 qio_task_new Task new task=0x56135cd66eb0 source=0x56135d1d7c00 func=0x56135af746c0 opaque=0x56135bf06400
22806@1547735341399865943 qio_task_thread_start Task thread start task=0x56135cd66eb0 worker=0x56135af72e50 opaque=0x56135c071d70
22806@1547735341399976816 qio_task_thread_run Task thread run task=0x56135cd66eb0
The second argument to this tool is the QEMU binary filename to be traced, which can be relative (to search $PATH) or absolute. What is clever is that it will set the SYSTEMTAP_TAPSET env variable to point to the right location to find the corresponding tapset definition. This is very useful when you have multiple copies of QEMU on the system and need to make sure systemtap traces the right one.
The ‘qemu-trace-stap‘ script takes a verbose arg so you can understand what it is running behind the scenes:
$ qemu-trace-stap run /home/berrange/usr/qemu-git/bin/qemu-system-x86_64 'qio*'
Using tapset dir '/home/berrange/usr/qemu-git/share/systemtap/tapset' for binary '/home/berrange/usr/qemu-git/bin/qemu-system-x86_64'
Compiling script 'probe qemu.system.x86_64.log.qio* {}'
Running script, <Ctrl>-c to quit
...trace output...
It can enable multiple probes at once
$ qemu-trace-stap run qemu-system-x86_64 'qio*' 'qcrypto*' 'buffer*'
By default it monitors all existing running processes and all future launched proceses. This can be restricted to a specific PID using the –pid arg
$ qemu-trace-stap run --pid 2532 qemu-system-x86_64 'qio*'
Finally if you can’t remember what probes are valid it can tell you
$ qemu-trace-stap list qemu-system-x86_64
ahci_check_irq
ahci_cmd_done
ahci_dma_prepare_buf
ahci_dma_prepare_buf_fail
ahci_dma_rw_buf
ahci_irq_lower
...snip...
This new functionality merged into QEMU upstream a short while ago and will be included in the QEMU 4.0 release coming at the end of April.
A couple of releases back I completely re-structured all the RPC handling code inside libvirt to make sure it could be properly shared between the client and server, as well as decoupling the RPC handling code from the implementation of the RPC functions. As part of this work I introduced a fairly comprehensive set of DTrace static probe points into the libvirt RPC code. While one could write a WireShark plugin that is able to decode the libvirt RPC protocol (oh look Michal already has written one), that would not be able to examine encrypted libvirt connections – which is pretty much all of them. By using static probes in the libvirt RPC code we can see the RPC messages being sent and received before/after encryption has been applied.
The observant will notice that I said I inserted DTrace static probes, while this blog subject line says SystemTAP. Well the SystemTAP developers had the good sense to make their userspace probing infrastructure support the DTrace static probe marker syntax. So inserting DTrace static probes into userspace code, trivially enables support for both DTrace and SystemTAP. I previously added DTrace probe support to QEMU/KVM and was very happy when Bryan Cantrill told me (at the recent KVM forum) that the DTrace probe support I added to KVM only needed minor build system tweaks to work on Solaris, despite my only ever having tested with Linux + SystemTAP.
Along with adding the DTrace markers to the libvirt RPC code, I also created two SystemTAP tapset files to make it simpler to use the probes from SystemTAP scripts. The first, /usr/share/systemtap/tapset/libvirt_probe.stp, contains the actual probe points, grouped by functional area, while the second, /usr/share/systemtap/tapset/libvirt_functions.stp, contains a bunch of helper functions for converting enum values into human friendly strings. The idea is that instead of seeing “Procedure 53”, a sysadmin would much rather see “Procedure domain_dump_core”. I won’t go into detail about what is in those two files here, instead I’ll just illustrate their use
Tracing the RPC client
Lets says we first want to see what messages the client is sending and receiving. There are two interesting probes here, “libvirt.rpc.server_client_msg_rx” and “libvirt.rpc.server_client_msg_tx_queue“. The former is triggered when a complete RPC message has been read off the wire, while the latter is triggered when an RPC message is queued for transmission. Ideally we would also have another probe triggered when an RPC message has been completely transmitted – that’s a future todo item. Simple usage of these two probes would be
# cat > demo.stp <<EOF
probe libvirt.rpc.client_msg_rx {
printf("client=%p len=%d program=%d version=%d procedure=%d type=%d status=%d serial=%d\n",
client, len, prog, vers, proc, type, status, serial);
}
probe libvirt.rpc.client_msg_tx_queue {
printf("client=%p len=%d program=%s version=%d procedure=%s type=%s status=%d serial=%d\n",
client, len, prog, vers, proc, type, status, serial);
}
EOF
# stap demo.stp
client=0x7f827c3b1010 len=28 program=536903814 version=1 procedure=66 type=0 status=0 serial=0
client=0x7f827c3b1010 len=36 program=536903814 version=1 procedure=66 type=1 status=0 serial=0
client=0x7f827c3b1010 len=40 program=536903814 version=1 procedure=1 type=0 status=0 serial=1
client=0x7f827c3b1010 len=28 program=536903814 version=1 procedure=1 type=1 status=0 serial=1
client=0x7f827c3b1010 len=28 program=536903814 version=1 procedure=110 type=0 status=0 serial=2
client=0x7f827c3b1010 len=48 program=536903814 version=1 procedure=110 type=1 status=0 serial=2
client=0x7f827c3b1010 len=28 program=536903814 version=1 procedure=2 type=0 status=0 serial=3
client=0x7f827c3b1010 len=28 program=536903814 version=1 procedure=2 type=1 status=0 serial=3
The example shows the results of running “virsh domname vm1”. There are 4 RPC calls made here, 66 (authenticate), 1 (open), 110 (get uri), 2 (close).
Tracing the client with friendly output
Unless you have memorized libvirt RPC enums, this isn’t a very friendly way to trace the code. This is where the aforementioned libvirt_functions.stp tapset comes into play.
# cat > demo.stp <<EOF
probe libvirt.rpc.client_msg_rx {
printf("R client=%p len=%d program=%s version=%d procedure=%s type=%s status=%s serial=%d\n",
client, len,
libvirt_rpc_program_name(prog, 0),
vers,
libvirt_rpc_procedure_name(prog, vers, proc, 0),
libvirt_rpc_type_name(type, 0),
libvirt_rpc_status_name(status, 0),
serial);
}
probe libvirt.rpc.client_msg_tx_queue {
printf("T client=%p len=%d program=%s version=%d procedure=%s type=%s status=%s serial=%d\n",
client, len,
libvirt_rpc_program_name(prog, 0),
vers,
libvirt_rpc_procedure_name(prog, vers, proc, 0),
libvirt_rpc_type_name(type, 0),
libvirt_rpc_status_name(status, 0),
serial);
}
EOF
# stap demo.stp
T client=0x7f3e3dec0010 len=28 program=remote version=1 procedure=auth_list type=call status=ok serial=0
R client=0x7f3e3dec0010 len=36 program=remote version=1 procedure=auth_list type=reply status=ok serial=0
T client=0x7f3e3dec0010 len=40 program=remote version=1 procedure=open type=call status=ok serial=1
R client=0x7f3e3dec0010 len=28 program=remote version=1 procedure=open type=reply status=ok serial=1
T client=0x7f3e3dec0010 len=28 program=remote version=1 procedure=get_uri type=call status=ok serial=2
R client=0x7f3e3dec0010 len=48 program=remote version=1 procedure=get_uri type=reply status=ok serial=2
T client=0x7f3e3dec0010 len=28 program=remote version=1 procedure=close type=call status=ok serial=3
R client=0x7f3e3dec0010 len=28 program=remote version=1 procedure=close type=reply status=ok serial=3
Much more friendly !
Tracing the server at the same time
It might desirable to see when the server itself receives the message, independently of when the client transmitted it. There are an identical set of probes available in the server, just replace ‘client’ with ‘server_client’ in the above examples. Thus the demo script can trivially be extended to show server messages at the same time:
# cat >> demo.stp << EOF
probe libvirt.rpc.server_client_msg_rx {
printf("R server=%p len=%d program=%s version=%d procedure=%s type=%s status=%s serial=%d\n",
client, len,
libvirt_rpc_program_name(prog, 0),
vers,
libvirt_rpc_procedure_name(prog, vers, proc, 0),
libvirt_rpc_type_name(type, 0),
libvirt_rpc_status_name(status, 0),
serial);
}
probe libvirt.rpc.server_client_msg_tx_queue {
printf("T server=%p len=%d program=%s version=%d procedure=%s type=%s status=%s serial=%d\n",
client, len,
libvirt_rpc_program_name(prog, 0),
vers,
libvirt_rpc_procedure_name(prog, vers, proc, 0),
libvirt_rpc_type_name(type, 0),
libvirt_rpc_status_name(status, 0),
serial);
}
# stap demo.stp
T client=0x7ff3c4855010 len=28 program=remote version=1 procedure=auth_list type=call status=ok serial=0
R server=0x17a2070 len=28 program=remote version=1 procedure=auth_list type=call status=ok serial=0
T server=0x17a2070 len=36 program=remote version=1 procedure=auth_list type=reply status=ok serial=0
R client=0x7ff3c4855010 len=36 program=remote version=1 procedure=auth_list type=reply status=ok serial=0
T client=0x7ff3c4855010 len=40 program=remote version=1 procedure=open type=call status=ok serial=1
R server=0x17a2070 len=40 program=remote version=1 procedure=open type=call status=ok serial=1
T server=0x17a2070 len=28 program=remote version=1 procedure=open type=reply status=ok serial=1
R client=0x7ff3c4855010 len=28 program=remote version=1 procedure=open type=reply status=ok serial=1
T client=0x7ff3c4855010 len=28 program=remote version=1 procedure=get_uri type=call status=ok serial=2
R server=0x17a2070 len=28 program=remote version=1 procedure=get_uri type=call status=ok serial=2
T server=0x17a2070 len=48 program=remote version=1 procedure=get_uri type=reply status=ok serial=2
R client=0x7ff3c4855010 len=48 program=remote version=1 procedure=get_uri type=reply status=ok serial=2
T client=0x7ff3c4855010 len=28 program=remote version=1 procedure=close type=call status=ok serial=3
R server=0x17a2070 len=28 program=remote version=1 procedure=close type=call status=ok serial=3
T server=0x17a2070 len=28 program=remote version=1 procedure=close type=reply status=ok serial=3
R client=0x7ff3c4855010 len=28 program=remote version=1 procedure=close type=reply status=ok serial=3
If the server is running on a different host than the client, just copy the demo.stp script to the other host and run a second copy there.
Further extensions
There are many further improvements that can be made to this script
- Display a timestamp on each message
- Associate each server side message with an individual socket
- Display payload length
- Display a message when the script is actually ready to run
To simplify life, we are maintaining a nice feature demonstration of the RPC SystemTAP probes in the libvirt GIT repository in the
examples/systemtap/rpc-monitor.stp file.
Here is what it can print out
0.000 begin
2.632 C + 0x7f1ea57dc010 local=127.0.0.1;0 remote=127.0.0.1;0
2.632 C > 0x7f1ea57dc010 msg=remote.1.auth_list(call, ok, 0) len=28
2.632 + S 0x1c1f710 local=127.0.0.1;0 remote=127.0.0.1;0
2.632 > S 0x1c1f710 msg=remote.1.auth_list(call, ok, 0) len=28
2.633 < S 0x1c1f710 msg=remote.1.auth_list(reply, ok, 0) len=36
2.633 C < 0x7f1ea57dc010 msg=remote.1.auth_list(reply, ok, 0) len=36 2.633 C > 0x7f1ea57dc010 msg=remote.1.open(call, ok, 1) len=40
2.633 > S 0x1c1f710 msg=remote.1.open(call, ok, 1) len=40
2.639 < S 0x1c1f710 msg=remote.1.open(reply, ok, 1) len=28
2.639 C < 0x7f1ea57dc010 msg=remote.1.open(reply, ok, 1) len=28 2.639 C > 0x7f1ea57dc010 msg=remote.1.get_uri(call, ok, 2) len=28
2.639 > S 0x1c1f710 msg=remote.1.get_uri(call, ok, 2) len=28
2.639 < S 0x1c1f710 msg=remote.1.get_uri(reply, ok, 2) len=48
2.640 C < 0x7f1ea57dc010 msg=remote.1.get_uri(reply, ok, 2) len=48 2.640 C > 0x7f1ea57dc010 msg=remote.1.domain_lookup_by_id(call, ok, 3) len=32
2.640 > S 0x1c1f710 msg=remote.1.domain_lookup_by_id(call, ok, 3) len=32
2.640 < S 0x1c1f710 msg=remote.1.domain_lookup_by_id(reply, error, 3) len=180
2.641 C < 0x7f1ea57dc010 msg=remote.1.domain_lookup_by_id(reply, error, 3) len=180 2.641 C > 0x7f1ea57dc010 msg=remote.1.close(call, ok, 4) len=28
2.641 > S 0x1c1f710 msg=remote.1.close(call, ok, 4) len=28
2.641 < S 0x1c1f710 msg=remote.1.close(reply, ok, 4) len=28
2.641 C < 0x7f1ea57dc010 msg=remote.1.close(reply, ok, 4) len=28
2.641 C - 0x7f1ea57dc010 local= remote=
2.641 - S 0x1c1f710 local=127.0.0.1;0 remote=127.0.0.1;0
Tracing other areas of libvirt code
The RPC code is not the only place with SystemTAP/DTrace probe markers in libvirt. We have also instrumented our main event loop and provide an examples/systemtap/events.stp demo that prints out info like this
0.000 begin
2.359 18185 + handle 1 4 1
2.360 18185 + handle 2 6 1
2.360 18185 * handle 2 0
2.360 14370 > handle 3 1
2.360 14370 + handle 33 16 1
2.361 14370 ~ 7 -1
2.361 14370 > handle 33 1
2.361 14370 * handle 33 1
2.361 14370 * handle 33 1
2.361 14370 * handle 33 3
2.361 14370 ~ 7 -1
2.361 14370 > handle 1 1
2.361 14370 ~ 7 -1
2.361 14370 > handle 33 2
2.361 14370 * handle 33 1
2.361 14370 ~ 7 -1
2.361 18185 * handle 2 1
2.362 18185 * handle 2 0
And finally we have instrumented our code which talks to the QEMU monitor, again providing a demo examples/systemtap/qemu-monitor.stp which prints out info like this
0.000 begin
3.848 ! 0x7f2dc00017b0 {"timestamp": {"seconds": 1319466931, "microseconds": 187755}, "event": "SHUTDOWN"}
5.773 > 0x7f2dc0007960 {"execute":"qmp_capabilities","id":"libvirt-1"}
5.774 < 0x7f2dc0007960 {"return": {}, "id": "libvirt-1"} 5.774 > 0x7f2dc0007960 {"execute":"query-commands","id":"libvirt-2"}
5.777 < 0x7f2dc0007960 {"return": [{"name": "quit"}, {"name": ....snip.... 5.777 > 0x7f2dc0007960 {"execute":"query-chardev","id":"libvirt-3"}
5.778 < 0x7f2dc0007960 {"return": [{"filename": ....snip.... 5.779 > 0x7f2dc0007960 {"execute":"query-cpus","id":"libvirt-4"}
5.780 < 0x7f2dc0007960 {"return": [{"current": true, "CPU": 0, "pc": 1048560, "halted": false, "thread_id": 13299}], "id": "libvirt-4"} 5.780 > 0x7f2dc0007960 {"execute":"set_password","arguments":{"protocol":"vnc","password":"123456","connected":"keep"},"id":"libvirt-5"}
5.782 < 0x7f2dc0007960 {"return": {}, "id": "libvirt-5"} 5.782 > 0x7f2dc0007960 {"execute":"expire_password","arguments":{"protocol":"vnc","time":"never"},"id":"libvirt-6"}
5.783 < 0x7f2dc0007960 {"return": {}, "id": "libvirt-6"} 5.783 > 0x7f2dc0007960 {"execute":"balloon","arguments":{"value":224395264},"id":"libvirt-7"}
5.785 < 0x7f2dc0007960 {"return": {}, "id": "libvirt-7"} 5.785 > 0x7f2dc0007960 {"execute":"cont","id":"libvirt-8"}
5.789 ! 0x7f2dc0007960 {"timestamp": {"seconds": 1319466933, "microseconds": 129980}, "event": "RESUME"}
5.789 < 0x7f2dc0007960 {"return": {}, "id": "libvirt-8"}
7.537 ! 0x7f2dc0007960 {"timestamp": {"seconds": 1319466934, "microseconds": 881214}, "event": "SHUTDOWN"}
Conclusion
The introduction of static probes into the libvirt code has been enormously helpful in understanding the operation of libvirt. While we have comprehensive debug logging present in libvirt is it hard to tailor the output to show the precise data desired. Traditional debuggers like GDB are not very practical when trying to understand the live operation of a heavily multi-threaded system crossing multiple processes, and while strace is useful in some scenarios it is too low level to be useful in most scenarios. SystemTAP userspace probing provides the kind of debugging experience / tool that really suits understanding the complex interactions in a system like libvirt. It is no co-incidence that the first set of probes we have written have focused on the libvirt event loop, RPC code and QEMU monitor – three of the areas in libvirt which are both very critical operationally, and exceptionally hard to debug with traditional approaches. We will certainly be expanding our use of static probe markers in systemtap in the future. My real immediate wishlist is for systemtap to get better at providing userspace stack traces, since it fails to provide a useful trace far too often, as compared to GDB.
Update: Mark Wielaard showed me what I had todo to get nice stack traces from SystemTAP. Apparently it is not getting enough memory space to deal with stack traces with its default settings. Telling it to use a little more memory makes it work nicely:
# cat > demo.stp <<EOF
probe libvirt.rpc.client_msg_rx {
printf("client=%p len=%d program=%d version=%d procedure=%d type=%d status=%d serial=%d\n",
client, len, prog, vers, proc, type, status, serial);
print_ustack(ubacktrace())
}
# stap -DTASK_FINDER_VMA_ENTRY_ITEMS=7680 demo.stp
client=0x7f775cf62010 len=36 program=536903814 version=1 procedure=66 type=1 status=0 serial=0
0x3c57f0b3dd : virNetClientIOHandleInput+0x87d/0x890 [/usr/lib64/libvirt.so.0.9.7]
0x3c57f0b9a0 : virNetClientIOEventLoop+0x5b0/0x630 [/usr/lib64/libvirt.so.0.9.7]
0x3c57f0cb23 : virNetClientSend+0x2b3/0x590 [/usr/lib64/libvirt.so.0.9.7]
0x3c57f0d47c : virNetClientProgramCall+0x26c/0x8a0 [/usr/lib64/libvirt.so.0.9.7]
0x3c57ef091e : callWithFD+0xce/0x120 [/usr/lib64/libvirt.so.0.9.7]
0x3c57ef099c : call+0x2c/0x40 [/usr/lib64/libvirt.so.0.9.7]
0x3c57efee80 : doRemoteOpen+0x890/0x20f0 [/usr/lib64/libvirt.so.0.9.7]
0x3c57f0341b : remoteOpen+0x9b/0x290 [/usr/lib64/libvirt.so.0.9.7]
0x3c57ec2133 : do_open+0x1f3/0x1100 [/usr/lib64/libvirt.so.0.9.7]
0x3c57ec4616 : virConnectOpenAuth+0x76/0xb0 [/usr/lib64/libvirt.so.0.9.7]
0x40ceb1 [/usr/bin/virsh+0xceb1/0x40000]
client=0x7f775cf62010 len=28 program=536903814 version=1 procedure=1 type=1 status=0 serial=1
0x3c57f0b3dd : virNetClientIOHandleInput+0x87d/0x890 [/usr/lib64/libvirt.so.0.9.7]
0x3c57f0b9a0 : virNetClientIOEventLoop+0x5b0/0x630 [/usr/lib64/libvirt.so.0.9.7]
0x3c57f0cb23 : virNetClientSend+0x2b3/0x590 [/usr/lib64/libvirt.so.0.9.7]
0x3c57f0d47c : virNetClientProgramCall+0x26c/0x8a0 [/usr/lib64/libvirt.so.0.9.7]
0x3c57ef091e : callWithFD+0xce/0x120 [/usr/lib64/libvirt.so.0.9.7]
0x3c57ef099c : call+0x2c/0x40 [/usr/lib64/libvirt.so.0.9.7]
0x3c57eff57a : doRemoteOpen+0xf8a/0x20f0 [/usr/lib64/libvirt.so.0.9.7]
0x3c57f0341b : remoteOpen+0x9b/0x290 [/usr/lib64/libvirt.so.0.9.7]
0x3c57ec2133 : do_open+0x1f3/0x1100 [/usr/lib64/libvirt.so.0.9.7]
0x3c57ec4616 : virConnectOpenAuth+0x76/0xb0 [/usr/lib64/libvirt.so.0.9.7]
0x40ceb1 [/usr/bin/virsh+0xceb1/0x40000]
....
This makes me very happy :-)