Avatar (Fabio Alessandro Locati|Fale)'s blog

libVirt on Hetzner

April 26, 2017

After many years of using Hetzner as a server provider, and having rented from them multiple servers for many reasons, I decided to rent a server with 128Gb of RAM to do some tests with many (virtualized) machines on top of CentOS.

As it often happens, hosting providers put in place a lot of security measurements that sometimes make doing simple stuff more complex. The first approach I tried was using the (only) Ethernet interface as a bridged interface, but that did not brought me very far. Speaking with the support they pointed out that it was impossible in my setup, so I moved to the second option: the broute.

In the broute approach, a bridge is added to the interface, but all the traffic gets routed. With the broute the configuration is very easy, in fact, to install and properly configure libVirt in such environment, the following steps are enough.

Let’s start installing the needed software.

yum install bridge-utils libvirt libvirt-daemon-kvm qemu-kvm virt-install libguestfs-tools

In my case, I’ll use qemu-kvm as virtualisation engine, and that’s why I’m installing it. Also, I install virt-install and libguestfs-tools since I want to do some changes to the images before running them.

The second step is to start and enable the libvirtd daemon:

systemctl enable libvirtd
systemctl start libvirtd

We can now move to configure the networking. In my case, I had the following situation IP wide:

My goal was to create 3 different networks:

Prepare the OS

The first step is ensure that the kernel rp_filter will not block the packages during the routing process. To do so, you need to put in the file /etc/sysctl.d/98-rp-filter.conf:

net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

In case you also want to be able to use nested-virtualisation acceleration, you need to put in /etc/modprobe.d/kvm_intel.conf:

options kvm-intel nested=1
options kvm-intel enable_shadow_vmcs=1
options kvm-intel enable_apicv=1
options kvm-intel ept=1

Even if it’s not mandatory, I always suggest to upgrade all your packages. To do so, you can run:

yum update -y

At this point you should reboot your box to ensure that all kernel options are properly loaded and that you are running the latest kernel version.

To make IPv6 work, it’s necessary to change the IPv6 class in /etc/sysconfig/network-scripts/ifcfg-enp0s2 (or the file that identifies your primary network interface) from /64 to /128. This will allow the bridge to obtain all the other IPv6 available in that class and be able to assign them to your virtual machines. To configure the three networks, we are going to use virsh, using xml files that describe the network.

Network creation

The file that describes the internal network is going to be called /tmp/internal.xml and contains:

<network>
  <name>internal</name>
  <bridge name="br-internal" stp="on" delay="0"/>
  <ip address="10.0.0.1" netmask="255.255.255.0">
  </ip>
</network>

As you can see we are just declaring an IP range (10.0.0.1/24), the bridge name (br-internal) and the network name (internal). We can now create the network:

virsh net-define /tmp/internal.xml
virsh net-start internal
virsh net-autostart internal

The second network is in the file /tmp/public.xml and contains the following:

<network>
  <name>public</name>
  <bridge name="br-public" />
  <forward mode="route"/>
  <ip address="88.99.247.224" netmask="255.255.255.240">
  </ip>
  <ip family="ipv6" address="2a01:4f8:10a:390a::1" prefix="64">
  </ip>
</network>

This is very similar to the previous one, with a couple of differences:

We can now create the network:

virsh net-define /tmp/public.xml
virsh net-start public
virsh net-autostart public

The third file, called /tmp/private.xml contains:

<network>
  <name>private</name>
  <bridge name="br-private" stp="on" delay="0"/>
  <forward mode="nat">
    <nat>
      <port start="1024" end="65535"/>
    </nat>
  </forward>
  <ip address="10.0.1.1" netmask="255.255.255.0">
  </ip>
</network>

This is very similar to the internal one, with the addition of the forward section, where we declare the ability of forwarding packages toward the internet via NAT.

We can now create the network:

virsh net-define /tmp/private.xml
virsh net-start private
virsh net-autostart private

If you now run virsh net-list you can now see the networks:

 Name                 State      Autostart     Persistent
----------------------------------------------------------
 defaul               active     yes           yes
 internal             active     yes           yes
 public               active     yes           yes
 private              active     yes           yes

In case you want to remove the default one (mainly for cleaness reasons):

virsh net-destroy default
virsh net-undefine default

Create the virtual machine

We can now move to starting our first virtual machine. I already had in /var/lib/libvirt/images/rhel-guest-image-7.3-35.x86_64.qcow2 a qcow2 RHEL7 image. The first step is copy it to a new file that will become the disk of the VM:

cp /var/lib/libvirt/images/rhel-guest-image-7.3-35.x86_64.qcow2 /var/lib/libvirt/images/vm00.qcow2

At this point we can customize the image with virt-customize to make the machine able to run properly:

virt-customize -a /var/lib/libvirt/images/vm00.qcow2 \
    --root-password password:PASSWORD_HERE \
    --ssh-inject root:file:/root/.ssh/fale.pub \
    --run-command 'systemctl disable cloud-init; systemctl mask cloud-init' \
    --run-command "echo -e 'DEVICE=eth0\nONBOOT=yes\nBOOTPROTO=none\nIPADDR=88.99.247.225\nNETMASK=255.255.255.255\nSCOPE=\"peer 88.99.208.11\"\nGATEWAY=88.99.208.11\nIPV6INIT=yes\nIPV6ADDR=2a01:4f8:10a:390a::10/64\nIPV6_DEFAULTGW=2a01:4f8:10a:390a::2' > /etc/sysconfig/network-scripts/ifcfg-eth0" \
    --run-command "echo -e 'DEVICE=eth1\nONBOOT=yes\nBOOTPROTO=none\nIPADDR=10.0.1.10\nNETMASK=255.255.255.0' > /etc/sysconfig/network-scripts/ifcfg-eth1" \
    --selinux-relabel

With this command we are going to perform a lot of changes. In order:

At this point we can create the machine with:

virt-install
    --ram 8192
    --vcpus=4
    --os-variant rhel7
    --disk path=/var/lib/libvirt/images/vm00.qcow2,device=disk,bus=virtio,format=qcow2
    --import
    --noautoconsole
    --network network:public
    --network network:private
    --name vm00

You can now connect via SSH to the machine.