Portrait of Martijn

CloudStack 4.4 Single Server on Ubuntu 14.04.1 with KVM

23 Feb 2015

Here are my notes for a single-server CloudStack 4.4.2 install on Ubuntu 14.04.1

Introduction

Back in 2013 I posted my notes on CloudStack Single Server on Ubuntu with KVM. I have had feedback from folks saying they found it useful, and asking for an update. This week I finally got round to doing a new CloudStack installation, so I reviewed that post, noticed some changes, and ran into some new issues. Some of the patches no longer applied cleanly, and I've replaced those with edit commands. Here are my updated notes.

A note of caution: this is only intended to allow people to experiment with CloudStack. For production systems you'll need to do more planning, have a different network layout, and probably pay some more attention to security apects (alternative to root password logins, use SSL). And you could consider going the CentOS/XenServer route, which appears to be the main target platform.

And of course if you misconfigure things, you can cause havoc with your network.

Cloudstack Documentation

The latest docs are reachable from the "Documentation" menu on the Cloudstack project page, including Installation docs for the latest version (4.4). You can use the menu at the bottom-left to get other versions and formats such as the PDF. There is a "Quick Installation Guide for CentOS" section, but not for Ubuntu/KVM. There is a KVM Hypervisor Host Installation section that covers CentOS and Ubuntu. I'll merge and cherry-pick from these docs to optimise and simplify for the Ubuntu single-server installation.

This time I'm installing on Ubuntu 14.04.1 LTS, on my on of my mini-itx hosts. I'll do a single-server install on a machine named "trinity10" with IP address 192.168.77.10. The hosts have a stock Ubuntu install on it, installed from a USB stick with UNetbootin, with only openssh-server added.

The commands below should be run as root.

Inital sever configuration

Per Prepare the Operating System, set our FQDN hostname, and use the actual IP address, not Ubuntu's annoying 127.0.1.1:

HOSTNAME=trinity10
DOMAINNAME=lab.stalworthy.net
IP=`ip addr show eth0 | grep 'inet ' | awk '{print $2}' | sed -e 's,/.*,,'`
sed -i -e "s/^127.0.1.1.*/$IP $HOSTNAME.$DOMAINNAME $HOSTNAME/"  /etc/hosts
cat /etc/hosts
hostname --fqdn
dig `hostname --fqdn` @8.8.8.8

And install NTP:

apt-get install openntpd

Also set a root password:

passwd root

And configure to allow ssh login with a password:

sed -i '/PermitRootLogin without-password/PermitRootLogin yes' /etc/ssh/sshd_config
service ssh restart

Next, configure a network bridge for its role as KVM hypervisor host, so that we don't have to interfere with networking half-way through the install. Adjust the IP addresses and netmasks for your environment:

apt-get install bridge-utils

cp /etc/network/interfaces /etc/network/interfaces.orig
cat >/etc/network/interfaces <<EOM
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

# Public network
auto cloudbr0
iface cloudbr0 inet static
    address 192.168.77.10
    netmask 255.255.255.0
    gateway 192.168.77.1
    dns-nameservers 192.168.77.1
    dns-domain lab.stalworthy.net
    bridge_ports eth0
    bridge_fd 5
    bridge_stp off
    bridge_maxwait 1

# Private network
auto cloudbr1
iface cloudbr1 inet manual
    bridge_ports none
    bridge_fd 5
    bridge_stp off
    bridge_maxwait 1
EOM

reboot

Now we install Cloudstack, which will pull in libvirt as a dependency. First Configure package repository:

cat >/etc/apt/sources.list.d/cloudstack.list <<EOM
deb http://cloudstack.apt-get.eu/ubuntu trusty 4.4
EOM
wget -O - http://cloudstack.apt-get.eu/release.asc|apt-key add -
apt-get update

Next Install the Management Server on the First Host:

apt-get --yes install cloudstack-management

Then Install the database server:

apt-get --yes install mysql-server

This will ask for a mysql password. I'll use mydbpasswd00.

cat >>/etc/mysql/conf.d/cloudstack.cnf <<EOM
[mysqld]
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=350
log-bin=mysql-bin
binlog-format = 'ROW'
EOM

service mysql restart

Now we need to setup the database. But here I ran into a problem, documented in CLOUDSTACK-8157. Basically, the setup script that ships with CloudStack is not compatible with the mysql version in this version of Ubuntu, so we need to patch it first:

cat >/tmp/CLOUDSTACK-8157.diff <<'EOM'
--- /usr/share/cloudstack-management/setup/create-schema-premium.sql.orig 2015-02-14 16:53:19.341079935 +0000
+++ /usr/share/cloudstack-management/setup/create-schema-premium.sql 2015-02-14 16:54:12.127456646 +0000
@@ -296,7 +296,7 @@
   `password` varchar(200) COMMENT 'password',
   `round_robin_marker` int COMMENT 'This marks the volume to be picked up for lun creation, RR fashion',
   PRIMARY KEY  (`id`),
-  CONSTRAINT `fk_netapp_volume__pool_id` FOREIGN KEY `fk_netapp_volume__pool_id` (`pool_id`) REFERENCES `netapp_pool` (`id`) ON DELETE CASCADE,
+  CONSTRAINT `fk_netapp_volume__pool_id` FOREIGN KEY `fk_netapp_volume__pool_id` (`pool_id`) REFERENCES `cloud`.`netapp_pool` (`id`) ON DELETE CASCADE,
   INDEX `i_netapp_volume__pool_id`(`pool_id`)
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8;

@@ -315,7 +315,7 @@
   `size` bigint NOT NULL COMMENT 'lun size',
   `volume_id` bigint unsigned NOT NULL COMMENT 'parent volume id',
   PRIMARY KEY (`id`),
-  CONSTRAINT `fk_netapp_lun__volume_id` FOREIGN KEY `fk_netapp_lun__volume_id` (`volume_id`) REFERENCES `netapp_volume` (`id`) ON DELETE CASCADE,
+  CONSTRAINT `fk_netapp_lun__volume_id` FOREIGN KEY `fk_netapp_lun__volume_id` (`volume_id`) REFERENCES `cloud`.`netapp_volume` (`id`) ON DELETE CASCADE,
   INDEX `i_netapp_lun__volume_id`(`volume_id`),
   INDEX `i_netapp_lun__lun_name`(`lun_name`)
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
EOM
cp /usr/share/cloudstack-management/setup/create-schema-premium.sql /usr/share/cloudstack-management/setup/create-schema-premium.sql.orig
    sudo apt-get install patch
patch -d / -p0 -u -i /tmp/CLOUDSTACK-8157.diff

Then run it (with your own passwords):

# mysql root password: mydbpasswd00
# cloud user password: mycupasswd11
# management_server_key: mymskey44
# database_key: mydbkey00
cloudstack-setup-databases cloud:mycupasswd11@localhost \
                --deploy-as=root:mydbpasswd00 \
                -e file \
                -m mymskey44 \
                -k mydbkey00

Prepare Storage

Per Prepare NFS Shares, prepare the primary and secondary NFS shares.

mkdir -p /export/primary /export/secondary

Install nfs:

apt-get install nfs-kernel-server

Export the /export directory:

cat >>/etc/exports <<EOM
/export  *(rw,async,no_root_squash,no_subtree_check)
EOM

exportfs -a

Configure NFS to run statd, on specific ports:

apt-get install nfs-common 
cp /etc/default/nfs-common /etc/default/nfs-common.orig
sed -i '/NEED_STATD=/ a NEED_STATD=yes' /etc/default/nfs-common
sed -i '/STATDOPTS=/ a STATDOPTS="--port 662 --outgoing-port 2020"' /etc/default/nfs-common
diff -du /etc/default/nfs-common.orig /etc/default/nfs-common

Configure lockd:

cat >> /etc/modprobe.d/lockd.conf <<EOM
options lockd nlm_udpport=32769 nlm_tcpport=32803
EOM

Restart NFS and test it shows the exported directory:

service nfs-kernel-server restart
# test:
showmount -e 127.0.0.1

Here I mount the NFS volumes locally. You need to do that on the management server so you can setup the System Template below.

IP=192.168.77.10
mkdir -p /mnt/primary /mnt/secondary
cat >>/etc/fstab <<EOM
$IP:/export/primary   /mnt/primary    nfs rsize=8192,wsize=8192,timeo=14,intr,vers=3,noauto  0   2
$IP:/export/secondary /mnt/secondary  nfs rsize=8192,wsize=8192,timeo=14,intr,vers=3,noauto  0   2
EOM
mount /mnt/primary
mount /mnt/secondary

Per Install and Configure libvirt, install cloudstack-agent.

apt-get install cloudstack-agent

cp /etc/libvirt/libvirtd.conf /etc/libvirt/libvirtd.conf.orig

sed -i '/#listen_tls = 0/ a listen_tls = 0' /etc/libvirt/libvirtd.conf
sed -i '/#listen_tcp = 1/ a listen_tcp = 1' /etc/libvirt/libvirtd.conf
sed -i '/#tcp_port = "16509"/ a tcp_port = "16509"' /etc/libvirt/libvirtd.conf
sed -i '/#auth_tcp = "sasl"/ a auth_tcp = "none"' /etc/libvirt/libvirtd.conf
diff -du /etc/libvirt/libvirtd.conf.orig /etc/libvirt/libvirtd.conf

Patch libvirt-bin.conf:

cp /etc/default/libvirt-bin /etc/default/libvirt-bin.orig
sed -i -e 's/libvirtd_opts="-d"/libvirtd_opts="-d -l"/' /etc/default/libvirt-bin
diff -du /etc/default/libvirt-bin.orig /etc/default/libvirt-bin

service libvirt-bin restart

Patch qemu.conf to listen on all interfaces:

cp /etc/libvirt/qemu.conf /etc/libvirt/qemu.conf.orig
sed -i '/# vnc_listen = "0.0.0.0"/ a vnc_listen = "0.0.0.0"' /etc/libvirt/qemu.conf
diff -du /etc/libvirt/qemu.conf.orig /etc/libvirt/qemu.conf

service libvirt-bin restart

Disable AppArmor:

ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/
ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd
apparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper

service libvirt-bin restart

We've already done the Configure the network bridges part.

From the Configure the firewall section, do the Open ports in Ubuntu part:

ufw allow proto tcp from any to any port 22
ufw allow proto tcp from any to any port 1798
ufw allow proto tcp from any to any port 16509
ufw allow proto tcp from any to any port 5900:6100
ufw allow proto tcp from any to any port 49152:49216

Now reboot and testing NFS is still OK:

reboot
rpcinfo -u 192.168.77.10 mount
showmount -e 192.168.77.10
mount /mnt/primary
mount /mnt/secondary

System Template Setup downloads the template that the Console and Storage Server VMs are based on:

/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \
    -m /mnt/secondary -u http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.1-7-kvm.qcow2.bz2 -h kvm -F

Check the UI is up, on http://192.168.77.10:8080/client. If you see 404s for /client/ and "The requested resource is not available" errors, try:

service cloudstack-management status
service cloudstack-agent status
service tomcat6 status

service cloudstack-management stop
service tomcat6 stop
service cloudstack-agent stop
ps -efl | grep java

service cloudstack-management start
service cloudstack-management status
service cloudstack-agent start
service cloudstack-agent status

Start using the UI

Now for Configuring your CloudStack Installation.

Log in to http://192.168.77.10:8080/client with username admin and password password.

Choose "Continue with basic installation". This will start a wizard that will walk through the configuration. You will need to adjust the network values for your environment, and make sure you use appropiate, free, ranges.

  • Add a new zone named "zone1", DNS1 192.168.77.1 and Internal DNS 192.168.77.1.
  • Add a new pod named "pod1", gateway 192.168.77.1, netmask 255.255.255.0, IP range 192.168.77.160-192.168.77.169.
  • Add a guest network, gateway 192.168.77.1, netmask 255.255.255.0, IP range 192.168.77.170-192.168.77.230.
  • Add a cluster named cluster1, Hypervisor KVM.
  • Add a host. Host Name "trinity10", user root, passsword for the root linux user.
  • Add primary storage: name primary1, protocol NFS, Scope Cluster, server 192.168.77.10, path /export/primary.
  • Add secondary storage: NFS server 192.168.77.10, path /export/secondary.
  • Hit Launch and pray.
  • This should go through a sequence of setup.

If this fails, you need have a chance to correct mistakes in the wizard.

If you go to http://192.168.77.10:8080/client, and click on Infrastructure, you should see: 1 zone, 1 pod, 1 cluster, 1 host, 1 primary storage, 1 secondary storage, and 2 System VMs. If you don't, then you won't be able to create VMs, and will need to debug/fix things first.

  • If you get get a 404 on http://192.168.77.10:8080/client, stop the management server and tomcat6, kill any leftover java process, and restart it.
  • If there is no Host, try adding it again, and doublecheck IP address and credentials.
  • If there is no primary storage, check NFS, check /mnt/primary is mounted and working, and re-add it.
  • If there is no secondary storage, check NFS, check /mnt/secondary is mounted and working, and re-add it.
  • As you go fixing these things, don't forget to hit Refresh on the Infrastructure page.
  • You should see two system VMs: one for managing secondary storage, and one for managing console access.
  • If they are missing, CloudStack will try to re-create them every 30 seconds or so.
  • If it's not re-creating check the zone is enabled (Infrastructure → Zones → zone1 → enable)
  • If it's trying but failing, then look at /var/log/cloudstack/management/management-server.log and /var/log/cloudstack/agent/agent.log.
  • Check the SSVM, templates, Secondary storage troubleshooting page for more tips. Some are out of date or wrong; for example, to ssh into the SSVM, use: ssh -i /var/lib/cloudstack/management/.ssh/id_rsa -p 3922

Register an ISO

Register an ISO: Templates → Select View ISO, Register ISO.

  • name: ubuntu-14.04.1-server-amd64
  • description: ubuntu-14.04.1-server-amd64
  • URL: http://releases.ubuntu.com/trusty/ubuntu-14.04.1-server-amd64.iso
  • Zones: all zones
  • OS: Ubuntu 12.04 (64-bit)
  • Bootable, Public, Featured

then click OK. Then click on the ISO row in the list, click the Zones tab, and check the Status shows x% Downloaded. Wait for the download to complete.

Setting up an Ubuntu Template

To make it easy to sping up fresh VMs, you'll want to make a Template. That's just a pre-installed linux image, with some special handling to have the hostname/network/password configuration happen on first boot.

Per Working with Templates, CloudStack ships with a CentOS template. The Creating a Linux Template instructions will let us create an Ubuntu template. The one gotcha there is that the sethostname script in the docs has bugs, at the time of writing. Some have already been fixed in the doc repositorys, but others introduced. I've fixed those, and submitted upstream. Until that gets fixed you can use my sethostname.