User Tools

Site Tools



This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
unix:virtualization:kvm:server [2020/10/16 23:19]
rodolico created
unix:virtualization:kvm:server [2022/05/19 02:13] (current)
Line 2: Line 2:
 **This is a work in progress, 20201015** **This is a work in progress, 20201015**
 +===== Location of Files =====
 +Just a note, the files created by virsh and virt-install are stored in **/​etc/​libvirt/​qemu/​**. However, they should not be hand-edited;​ only edit with virsh.
 ===== Install and Configure ===== ===== Install and Configure =====
Line 13: Line 17:
 You should see either vmx or svm in the output. You should see either vmx or svm in the output.
-Now, install the basic packages needed, and nothing else+Now, install the basic packages needed, ​a couple of utilities, but not all the extra crud. //​netcat-openbsd//​ is only needed if you're going to cluster ​and migrate virtuals from one hypervisor to another, from what I can tell. **virt-top** is a nice little //top// for seeing what is running and what resources they are using in real time.
 <code bash> <code bash>
-apt install -y --no-install-recommends qemu-kvm libvirt-clients libvirt-daemon-system bridge-utils libguestfs-tools genisoimage virtinst libosinfo-bin virt-top+apt install -y --no-install-recommends qemu-kvm libvirt-clients libvirt-daemon-system bridge-utils libguestfs-tools genisoimage virtinst libosinfo-bin virt-top ​netcat-openbsd
 reboot # brings libraries online reboot # brings libraries online
 </​code>​ </​code>​
Line 39: Line 43:
 ===== Defining Network ===== ===== Defining Network =====
 +==== Setting up bridges ====
 +For your network, you need bridges for the outside world. ​
 +=== Simple ===
 +This is a basic setup that will work for a single interface as per the Debian documentation. It sets up one bridge off of eth0 and gives it a static IP.
 +<code bash interfaces>​
 +auto lo
 +iface lo inet loopback
 +# The primary network interface
 +auto eth0
 +#make sure we don't get addresses on our raw device
 +iface eth0 inet manual
 +iface eth0 inet6 manual
 +#set up bridge and give it a static ip
 +auto br0
 +iface br0 inet static
 +        address
 +        netmask
 +        network
 +        broadcast
 +        gateway
 +        bridge_ports eth0
 +        bridge_stp off
 +        bridge_fd 0
 +        bridge_maxwait 0
 +        dns-nameservers
 +=== Real World ===
 +I'm hoping, if you're reading this article, you know how to set up bonding and vlans. The following ​ is a more real world scenario and sets up three bridges, br_lan, br_dmz and br_wan for LAN, DMZ and public interface respectively. br_wan and br_dmz are set to dummy IP's with a netmask of,​ meaning they can't respond to anything but themselves. br_lan gets its IP from DHCP, so only the LAN interface can be accessed on the hypervisor. Modify it as you want.
 <code bash interfaces>​ <code bash interfaces>​
Line 74: Line 117:
    ​vlan-raw-device bond0.30    ​vlan-raw-device bond0.30
 +# the public interface on vlan 10
 auto br_wan auto br_wan
 iface br_wan inet static iface br_wan inet static
Line 83: Line 127:
    ​bridge_maxwait 0    ​bridge_maxwait 0
 +# the DMZ on vlan 20
 auto br_dmz auto br_dmz
 iface br_dmz inet static iface br_dmz inet static
Line 92: Line 137:
         bridge_maxwait 0         bridge_maxwait 0
 +# the private (LAN) interface on vlan 30
 auto br_lan auto br_lan
 iface br_lan inet dhcp iface br_lan inet dhcp
Line 100: Line 146:
 </​code>​ </​code>​
 +==== Adding network to virt-lib ====
 +In order to use a network with vir-lib, you need to define it. The best way is to create a few XML files, then use virsh to define them into the system.
 +=== One at a time ===
 +Create one XML file per interface as follows:
 +<code xml br_wan.xml>​
 +  <​name>​br_wan</​name>​
 +  <forward mode="​bridge"/>​
 +  <bridge name="​br_wan"/>​
 +Then, import it into the system with virsh, then set it to autostart on boot
 +<code bash>
 +# import the network xml file
 +virsh net-define --file br_wan.xml
 +# set to autostart on boot
 +virsh net-autostart br_wan
 +=== Lazy Approach ===
 +I'm lazy, so I just created all three, then imported them all at one time.
 +<code bash>​
 +#! /​usr/​bin/​env bash
 +# create the xml definitions. The br is prepended. ​
 +# Add/remove interfaces if needed
 +for interface in wan lan dmz
 +cat << EOF > br_$interface.xml
 +       <​network>​
 +         <​name>​br_$interface</​name>​
 +         <​forward mode="​bridge"/>​
 +         <​bridge name="​br_$interface"/>​
 +       </​network>​
 +# uncomment this if you want to view your xml files but
 +# not process them
 +# exit
 +# find all xml files and do the net-define
 +for interface in `ls *.xml`
 +   virsh net-define --file $interface
 +# since the bridge name is followed by xml, simply remove that
 +# and set to autostart and start it
 +for interface in `ls *.xml | cut -d'​.'​ -f1`
 +   virsh net-autostart $interface ​
 +   virsh net-start $interface
 +# show me the list of network names
 +virsh net-list
 +This script assumes your network names are of the form br_//​something//​ and it creates the file name as br_//​something//​.xml. It then looks for all XML files (so, you don't want any others in the current directory), the processes them.
 +The last loop assumes there are no periods in the network name. Be warned.
 +===== Using Storage =====
 +In our example, we are going to use LVM2 to grab a piece of the disk for a new virtual. You can also use a file (File Backed Device, or FBD) by running //​fallocate//,​ or set up access to an iSCSI. virt-install will, by default, create an FBD in its default location, so if you're happy with that, ignore this whole section.
 +==== File Backed Device =====
 +By default, libvirt uses File Backed Devices (FBD'​s) from a pool defined internally. You can manually override this by defining a new pool, or by creating a file in your location using fallocate. For example, to create a 10G file in /srv/images named test.disk, you would use:
 +<​code>​fallocate -l 10G /​srv/​images/​test.disk</​code>​
 +and use that when you create the virtual. However, it is better (easier) if you define a pool (or use the default). I have not researched this; see //man virt-install//​.
 +==== LVM2 ====
 +Just create an LV the way you always do.
 +<code bash>​lvcreate -L 10G -n test.disk vg0</​code>​
 +==== iSCSI ====
 +Ok, if you're using iSCSI, I'm guessing you know how to set it up. Just make sure it is available, then use the correct path when you create the image.
 +===== Doing the Install =====
 +You can probably create a virtual by manually creating the XML file, but why do that when virt-install is your friend. Sure, there are a bunch of parameters, but they are very, very well documented, and will create your system for your rapidly
 +This example creates a virtual installing the opnSense firewall/​router.
 +<code bash>
 +virt-install \
 +   --hvm \
 +   ​--connect ​   qemu:///​system \
 +   ​--name ​      ​router-a \
 +   ​--memory ​    4096 \
 +   ​--vcpus ​     4 \
 +   ​--disk ​      ​path=/​dev/​vg0/​router-a.disk0,​bus=virtio,​target=sda \
 +   ​--graphics vnc,​port=5901 \
 +   ​--noautoconsole \
 +   ​--cdrom ​    /​media/​xen-store/​OPNsense-20.1-OpenSSL-dvd-amd64.iso \
 +   ​--os-variant freebsd11.1 \
 +   ​--metadata uuid=d9510e01-e461-461f-9aa8-3cee223cb4a0,​name=router-a,​title=router-a,​description='​Primary Router'​ \
 +   ​--boot ​    ​hd,​cdrom,​menu=on \
 +   ​--network ​ bridge=br_wan,​mac=00:​16:​3e:​bd:​26:​70,​model=virtio \
 +   ​--network ​ bridge=br_dmz,​mac=00:​16:​3e:​bd:​26:​71,​model=virtio \
 +   ​--network ​ bridge=br_lan,​mac=00:​16:​3e:​bd:​26:​72,​model=virtio
 +Other useful options are
 +  * --dry-run # don't really do it
 +  * --controller virtio-scsi # this is specific to KVM, but faster there
 +  * osinfo-query os # this shows the OS's available for the --os-variant flag
 +most of the parameters are self evident, but I'll quickly talk about why I did some of them.
 +  * --graphics vnc,​port=5901 - Our servers are headless and have no GUI. I want to be able to connect during install using VNC. By defining the port here, it is not auto-selected (and changeable) across boots
 +  * --noautoconsole - When starting the virtual, virt-lib tries to bring up a GUI for virtviewer to take over, or run virsh console to bring up a serial console. Since I don't want either, I disable it here
 +  * --os-variant freebsd11.1 - By defining this, virt-install can set up some defaults that work well with the known OS.
 +  * --metadata uuid=d9510e01-e461-461f-9aa8-3cee223cb4a0,​name=router-a,​title=router-a,​description='​Primary Router'​
 +    * title, name and description will show up in some reports. ​
 +    * uuid is good so the virtual does not get a new, randomly generated UUID every time it runs. Use the uuidgen command to get a randomly generated one for each new machine
 +===== Performing Installation =====
 +For most systems, I use VNC to do the install. Since I'm remote, I use the following ssh command:
 +<code bash>ssh -L localhost:​5910:​localhost:​5901 server</​code>​
 +  * 5910 is the local VNC port I'll attach to (ie, localhost:​5910)
 +  * 5901 is the VNC port I told virt-install to use
 +  * server is the name of the hypervisor
 +Then, when I run the virt-install command, I simply make a VNC connection to localhost:​5910 and can do my install.
 +===== Setting other parameters =====
 +If you want the domain to come up automatically when the hypervisor is turned on, the autostart flag needs to be set. Do this with <code bash>​virsh autostart domainname</​code>​
 +===== Links =====
 +  * https://​​faq/​install-kvm-server-debian-linux-9-headless-server/​
 +  * https://​​index.php/​Managing_KVM_on_RHEL_6_using_the_virsh_Command-line_Tool
 +  * https://​​virsh-commands-cheatsheet/​
 +  * https://​​questions/​350806/​convert-libvirt-xen-configuration-to-native
 +  * https://​​sources/​virshcmdref/​html/​
 +  * https://​​2016/​04/​how-to-migrate-your-virtual-box.html (migrate virtualbox images to kvm qcow2)
 +  * https://​​questions/​627238/​kvm-libvirt-how-to-configure-static-guest-ip-addresses-on-the-virtualisation-ho#​627245 (set sticky IP's on KVM)
unix/virtualization/kvm/server.1602908385.txt.gz ยท Last modified: 2020/10/16 23:19 by rodolico