Routed subnets without NAT for libvirt managed virtual machines in Fedora

Posted: December 13th, 2009 | Filed under: libvirt, Virt Tools | 6 Comments »

There are a huge number of ways of configuring networking for virtual machines when running libvirt. The two most common options, and our default recommendations for people, are either NAT (also known as “virtual networking”) or bridging (also known as “shared physical device”). The NAT option has the advantage that it can be made to work out of the box for pretty much all machines, even though with only wifi or dialup networking, but only allows outbound access from VMs, no incoming connections from outside the host. The bridging option has the advantage that machines on the wider LAN can access guests on the host, but it does not work with wifi.

This post is going to quickly describe a 3rd way of providing network connectivity to VMs, which we called the ‘routed’ option. In terms of its implementation in libvirt, it is actually just a variant on the NAT option, but without the NAT. For the purposes of this discussion I am going to describe my home network setup which is entirely wireless.

WLAN router
This is an LinkSys WRT54GL wireless router which of course runs Linux in the form of OpenWRT Kamikaze. This provides a DHCP service on the wireless LAN for the subnet 192.168.254.0/24
Mini server
This is a Mac Mini running Fedora 12, primarily acting as server for running SqueezeCenter. While it has an ethernet port, its location in the house means wifi access is the only option.
Random laptops
This is mostly the IBM Thinkpad I do most of my day-to-day work on. Again it only ever connects over wifi

The requirement is to run a number of virtual machines on the mini server, and be able to have unrestricted access to them from the laptop. Since the mini server is wireless, bridging is out of the question. Similarly, since I need unrestricted access to the VMs, the NAT option is also not viable. Hence this post about setting up routed networking.

Configuring the virtualization host

The libvirt virtual networking service normally gives you a ‘virbr0’ configured todo NAT. The XML format, however, allows you to specify that any virtual network be setup without NAT. It is perfectly acceptable to have many virtual networks on the same host, some using NAT, some not. Thus leave the default network alone, and simply define a new one using a subnet of 192.168.200.0/24 on the mini server.

# cat > vms.xml <<EOF
<network>
  <name>vms</name>
  <forward mode='route'/>
  <bridge name='virbr1' />
  <ip address='192.168.200.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.200.2' end='192.168.200.254' />
    </dhcp>
  </ip>
</network>
EOF

# virsh net-define vms.xml
Network vms defined from vms.xml

With the configuration for the network defined, it can be started, and indeed set to startup upon system boot

# virsh net-start vms
Network vms started

# virsh net-autostart vms
Network vms marked as autostarted

If you look at iptables output you will set libvirt has defined a series of iptables rules in the FORWARD chain to allow traffic to be pass from the virtual network to the LAN & vica-verca. You must, however, ensure that the sysctl net.ipv4.ip_forward is enabled otherwise the kernel won’t even attempt forwarding!

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 2856  203K ACCEPT     udp  --  virbr1 *       0.0.0.0/0            0.0.0.0/0           udp dpt:53
    0     0 ACCEPT     tcp  --  virbr1 *       0.0.0.0/0            0.0.0.0/0           tcp dpt:53
    2   656 ACCEPT     udp  --  virbr1 *       0.0.0.0/0            0.0.0.0/0           udp dpt:67
    0     0 ACCEPT     tcp  --  virbr1 *       0.0.0.0/0            0.0.0.0/0           tcp dpt:67


Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 453K  672M ACCEPT     all  --  *      virbr1  0.0.0.0/0            192.168.200.0/24
 245K   13M ACCEPT     all  --  virbr1 *       192.168.200.0/24     0.0.0.0/0
    0     0 ACCEPT     all  --  virbr1 virbr1  0.0.0.0/0            0.0.0.0/0
    0     0 REJECT     all  --  *      virbr1  0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable
    0     0 REJECT     all  --  virbr1 *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable

The 4 rules in the INPUT chain allow DHCP/DNS requests to the dnsmasq instance running on virbr1. The first rule in the FORWARD chain allows traffic from the hosts’s WLAN to pass to the VM subnet only. The second rule allows traffic from VMs to the WLAN. The third rule allows traffic between VMs. The final two rules block everything else, mostly to protect against IP spoofing

That really is all that is needed on the virtualization host to setup a routed network. The next step takes place on the LAN router.

Configuring the LAN/WLAN router

The virtualization host is now all set to forward traffic from 192.168.200.0/24 to & from the WLAN 192.168.254.0/24. This on its own though is not sufficient, because no other hosts on the WLAN know where the 192.168.200.0/24 subnet is ! It is thus necessary to configure a static route on the LAN/WLAN router, in this case my OpenWRT box.

To be able to route to the new subnet, the virtualization host needs to have a static IP address, even if it is being configured via DHCP. I fixed my virt host to use the IP 192.168.254.223, so now enabling route to the subnet containing the VMs merely requires adding one static route. Using the ‘ip’ command this could be done with:

# ip route add 192.168.254.0/24 via 192.168.254.223

OpenWRT Kamikaze of course comes with a config file that lets you do that in a way that is persistent across reboots.

# cat >> /etc/config/networks <<EOF
config 'route' 'minivms'
        option 'interface' 'lan'
        option 'target' '192.168.200.0'
        option 'netmask' '255.255.255.0'
        option 'gateway' '192.168.254.223'
EOF

# /etc/init.d/network restart

Depending on the precise way the network interfaces on the router are configured, and the current iptables setup it might be necessary to add a rule to the FORWARD chain. In my case I had to allow the 192.168.200.0 subnet to be forwarded over the ‘br-lan’ device, by adding to /etc/firewall.user

# iptables -I FORWARD 1 -i br-lan -o br-lan --dest 192.168.254.0/24 -j ACCEPT

With that in place my laptop can now ping guests guests running on the mini server’s virtual network, and vica-verca. No NAT or bridging involved and all playing nicely with wifi.

Configuring the guests

With the router and virtual host both configured the stage is set to provision virtual machines. If using virt-manager, then in the last step of the “New VM wizard”, expand the ‘Advanced options’ panel and simply select the ‘vms’ network from the drop down list of choices. The guest will then be connected to the newly defined virtual network, instead of the default NAT based one.
If provisioning using virt-install on the command line, then use the argument ‘–network network:vms’ to tell it to use the new virtual network.

Other ways of providing routed networking

While the above setup only requires three configuration steps, (1. define network on the virt host, 2. add static route on WLAN router. 3. add iptables rules), the obvious pain point here is that you might not have the ability to add static routes on the WLAN router. If that is not the case, then you could provide the static routes on all the client machines on the WLAN (ie add it to the laptop itself). This is sub-optimal too for rather obvious scalability reasons.

What we really want is to be able to provide routed networking without having to define a new IP subnet. In other words we need to figure out how to make libvirt’s virtual networking capability support Proxy ARP either of individual IPs or by subnetting. Patches on a postcard please… :-)

6 Responses to “Routed subnets without NAT for libvirt managed virtual machines in Fedora”

  1. Tony Stwart says:

    Very interesting follow through I will try it out when I get home. Thank you for the contribution.

  2. David Li says:

    Is it possible to use virsh-net* cmds to attach a VM to the new routed network not using virt-manager gui?

  3. Daniel Berrange says:

    You can use virsh edit $VMNAME to change the configuration of an existing VM’s network interfaces.

  4. Thank you for this clear tutorial. I have a copy of my website on a virtual machine on a laptop with ubuntu 12.04.3. My router is also a wrt54gl connecting my laptop and pc with wifi. In the past I used VirtualBox to host the virtual pc and it was not difficult to see mij copy website on all pc s and ipad in my home network. After som problems with Virtualbox on my laptop I decided to use Kvm/qemu with Virtual Machine Manager as gui.
    After converting my image it was not difficult to get it working, but I could my copy website only see on the laptop where the vm was hosted. I have tried several howtos some easier some more difficult but none working. My conclusion was that the problem was bridging in combination with a wireles network card.
    Seeing that you did it without bridging I gave it a try.
    It was even easier than I had expected. I did not need to edit the xml for the virtual network all the options to get the same xml are present in the Vmm Gui to create new virtual network.
    After that I added the net.ipv4.ip_forward =1 option to a new file in /etc/sysctl.d and restarted tje service with the command that I found in the README file in that directory.
    In my wireless router it was easy to add the rule via the basic/advanced routing option in the gui.

    After restarting the virtual machine my copy website was available on my laptop and my ipad.

    This solution not only works but is also more simple than most howto’s that I read before

  5. Marco says:

    Thank you, very interesting and useful.
    Just a note: I think you did two mistakes in the “Configuring the LAN/WLAN router” section, in the first and third shell snippets you have “192.168.254.0/24” where it should be instead “192.168.200.0/24” if I understand it correctly.

  6. yvan says:

    Hello.

    I am a new user of libvirtd.
    After using Nat network in VM, I am looking for a configuration with the bridged/routed network.
    So, I have created with virt-manager this network:
    (Hppro is my real server: it’s a laptop)

    root@hppro:~# virsh net-list
    Nom État Démarrage automatique Persistent
    ———————————————————-
    routed1 actif yes yes

    root@hppro:~# virsh net-dumpxml routed1

    routed1
    bda0ace1-10f0-4361-956a-baba0e5e38a0

    root@hppro:~# ifconfig virbr1
    virbr1 Link encap:Ethernet HWaddr 52:54:00:36:af:5f
    inet adr:192.168.100.1 Bcast:192.168.100.255 Masque:255.255.255.0
    root@hppro:~# ifconfig virbr1-nic
    virbr1-nic Link encap:Ethernet HWaddr 52:54:00:36:af:5f
    BROADCAST MULTICAST MTU:1500 Metric:1

    I don’t understand what virbr1-nic is.

    My lan is 192.168.2.0/24 and I have access to another hosts on it, and my router 192.168.2.1 is connected to the internet.

    Firewall rules added automaticaly by libvirtd, for virbr1 are ok.

    Here is my local configuration:
    root@hppro:~# ifconfig eth0
    eth0 Link encap:Ethernet
    inet adr:192.168.2.104 Bcast:192.168.2.255 Masque:255.255.255.0

    root@hppro:~# route -n
    Table de routage IP du noyau
    Destination Passerelle Genmask Indic Metric Ref Use Iface
    0.0.0.0 192.168.2.1 0.0.0.0 UG 0 0 0 eth0
    192.168.2.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0
    192.168.100.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr1

    root@hppro:~# brctl show virbr1
    bridge name bridge id STP enabled interfaces
    virbr1 8000.52540036af5f yes virbr1-nic
    vnet0


    I don’t understand your doc or this one:
    http://wiki.libvirt.org/page/Networking#Debian.2FUbuntu_Bridging .

    You seems directly change the virtual switch Ip of libvirtd: with an ip on your lan ?
    and adding a route.
    But the previous ip was a gateway for VMs under Libvirtd ?

    Where is the brige (I have seen we can’t make a bridge with wifi, but: if it was possible ?)

    In my case: if I add eth0 to the bridge: “brctl addif virbr1 eth0” (like the wiki seems do),
    eth0 is fixed : no access to the lan, no internet

    Without ethO in the bridge:
    I have network between the server (hppro) and VMs (I can ping each-others)
    (and I have access normally to the Lan and internet from Hppro)

    So I don’t understand all … ;)
    Help will be appreciate.
    Thanks!

Leave a Reply





Spam protection: Sum of n1ne plus t3n ?: