Thursday, June 07, 2012

Cloudstack Infrastructure Build and Deployment - Ubuntu 10.04

* This is one of the docs I wrote about deploying Cloudstack Build and Deployment on Ubuntu 10.04. There were some issues running it properly with Xen Hypervisor so I never fully completed or published this. In the meantime, Ubuntu 12.04 also came out and there's almost nothing to date with Cloudstack being updated with to work with 12.04. I will probably continue this at a later date.
Management Server
This install assumes the use of Ubuntu 10.04 and assumes you are logged in as root.

The Management Server in this example is installed on a vmware machine with 1 x 120 gig disk and 1 x 500 gig disk.

apt-get -y update
apt-get -y upgrade
apt-get install -y ntp nfs-kernel-server
mkdir /etc/exports.d

Also in this example we have a 1.5TB second drive setup at /dev/cciss/c1d0p1 do the following

fdisk /dev/cciss/c1d0
pvcreate /dev/cciss/c1d0p1

mkfs.ext4 /dev/cciss/c1d0p1

vim /etc/fstab

Add an entry to the bottom of /etc/fstab then restart the server

/dev/cciss/c1d0p1       /srv        ext4    defaults,auto,_netdev 0 0

Disable IPV6
vim /etc/sysctl.conf
Add the following line to the end of the file /etc/sysctl.conf (as root or use sudo)
net.ipv6.conf.all.disable_ipv6 = 1
Run the following command to make this change effective immediately

Restart the Computer

cd ~
mkdir downloads
cd downloads
tar -zxvf CloudStack-oss-3.0.1-1-ubuntu10.04.tar.gz
cd CloudStack-oss-3.0.1-1-ubuntu10.04

tar -zxvf CloudStack-oss-3.0.2-1-ubuntu10.04.tar.gz
cd CloudStack-oss-3.0.2-1-ubuntu10.04

(Choose the Upgrade Option)

mySQL Database
If you didn’t install mySQL, when you run ./ it will prompt you. Select “D”

Assuming mySQL installed open up /etc/mysql/my.cnf and add the following to the
[mysqld] section

binlog-format = 'ROW'
#default bind address is change to
bind-address =

Enable Root access from any machine
mysql -p
grant all on *.* to root@'%' identified by 'password';

Exit and restart mysql

service mysql restart

Install the Management server and choose “M” as the option


Login to mysql and create the cloud user and database

cloud-setup-databases cloud:cloudpassword@localhost --deploy-as=root:password

Choose M for Management Server Software. After the install is complete and done, restart the following services:

Make sure the NFS and RPCBind services are started

service nfs-kernel-server start

Run the command


Prepare the NFS Shares for Storage
mkdir -p /srv/export/primary
mkdir -p /srv/export/secondary

vim /etc/exports

Open up the /etc/exports and add the following

/srv/export *(rw,async,no_root_squash)

vim /etc/idmapd.conf

update the /etc/idmapd.conf

domain =

Show the exports
showmount -e
Download the System VM Templates

/usr/lib/cloud/agent/scripts/storage/secondary/cloud-install-sys-tmplt -m /srv/export/secondary -u -h kvm -F

/usr/lib/cloud/agent/scripts/storage/secondary/cloud-install-sys-tmplt -m /srv/export/secondary -u -h vmware -F

/usr/lib/cloud/agent/scripts/storage/secondary/cloud-install-sys-tmplt -m /srv/export/secondary -u -h xenserver -F

To Debug or View the system logs go to


These logs are very important to review as they will tell you exactly why or why you could not add a host to the cloud infrastructure

on a linux machine or ssh console use the tail command so you can view the log updates

tail /var/log/cloud/management/management-server.log -f
Host Server (Xen)
Xen allows the use of AMI (Amazon Machine Images) for compatibility

XenServer is actually a stand alone server product, for this release download 6.0.2 from. Down the iso, mount and install to a physical server. Be sure not to use VLAN for the management interface as Cloudstack doesn’t support this.

service nfs start
cd ~
mkdir downloads
cd downloads

tar -zxvf xenserver-cloud-supp.tgz

xe-install-supplemental-pack xenserver-cloud-supp.iso

Advanced Networking
xe-switch-network-backend openvswitch

Basic Networking
xe-switch-network-backend bridge

Then reboot

Test and Launch your instance

After you can launch an instance

default centOS Install

Importing an Amazon Machine Image

cd ~
cd downloads
mkdir ami
cd ami
mkdir -p /mnt/loop/ubuntu1110x64
tar -zxvf ubuntu-11.10-server-cloudimg-amd64.tar.gz
mount -o loop oneiric-server-cloudimg-amd64.img /mnt/loop/ubuntu1110x64/

Creating a new VM Template

You can download this script from or use the one below. Simply change the distro for newer ubuntu versions

Make sure the file below is executed on the Xenserver host conosle

cd ~
mkdir downloads
cd downloads
chmod 700
#In this example we will create and Ubuntu 10.0.4 LTS Template for XenServer
##, this creates Ubuntu server 10.04 32 and 64 bit templates on Xenserver 5.6. Net install only
## Author: David Markey <>
## This is not an officially supported guest OS on XenServer 5.6

LENNY=$(xe template-list name-label=Debian\ Lenny\ 5.0\ \(32-bit\) --minimal)

if [[ -z $LENNY ]] ; then
   echo "Cant find lenny 32bit template, is this on 5.6?"
   exit 1

distro="Ubuntu 10.04"
arches=("32-bit" "64-bit")

for arch in ${arches[@]} ; do
   echo "Attempting $distro ($arch)"
   if [[ -n $(xe template-list name-label="$distro ($arch)" params=uuid --minimal) ]] ; then
       echo "$distro ($arch)" already exists, Skipping

       NEWUUID=$(xe vm-clone uuid=$LENNY new-name-label="$distro ($arch)")
       xe template-param-set uuid=$NEWUUID other-config:install-methods=http,ftp \
        other-config:install-repository= \
        PV-args="-- quiet console=hvc0 partman/default_filesystem=ext3" \
        other-config:debian-release=lucid \

       if [[ "$arch" == "32-bit" ]] ; then
           xe template-param-set uuid=$NEWUUID other-config:install-arch="i386"
           xe template-param-set uuid=$NEWUUID other-config:install-arch="amd64"
       echo "Success"



echo "Done"
Installing Ubuntu 10.04 LTS

In this example, you will need to create the Ubuntu Virtual Machine Template. The instructions are at but I will also let them here

During the Install simply follow the screenshots below

Install XenTools
change to the root user

cd ~
mkdir /mnt/cdrom
mount /dev/cdrom /mnt/cdrom
dpkg -i /mnt/cdrom/Linux/xe-guest-utilities_6.0.2-756_amd64.deb

also make sure to set eth0 as DHCP in the /etc/network/interfaces file

auto eth0
iface eth0 inet dhcp

After this is done, shut down the virtual machine and connect to the console of the Xen server where you created the Virtual Machine.

You will also need a web server for this in order to serve the .vhd file to your management server where you are creating the template.

Root into your xen console server

xe vm-list

this will show you a list of the virtual machines.

Choose the UUID you just created. this example the command shows this output:

uuid ( RO)           : 2d6b4461-6c11-9491-1c34-b2b9914435ac
    name-label ( RW): Ubuntu 10.04 (64-bit) Template
   power-state ( RO): halted

Now run the command to list the storage associated with the template.

xe vbd-list vm-uuid=2d6b4461-6c11-9491-1c34-b2b9914435ac

This is the output

uuid ( RO)             : b10a26c2-2bca-f2fc-8bca-d8325b5ef873
         vm-uuid ( RO): 2d6b4461-6c11-9491-1c34-b2b9914435ac
   vm-name-label ( RO): Ubuntu 10.04 (64-bit) Template
        vdi-uuid ( RO): 2d4b29a1-07b8-4a26-9dd1-188bed075e0e
           empty ( RO): false
          device ( RO): xvda

uuid ( RO)             : 4ca40894-5464-1f29-e84d-e7ac0f139a8f
         vm-uuid ( RO): 2d6b4461-6c11-9491-1c34-b2b9914435ac
   vm-name-label ( RO): Ubuntu 10.04 (64-bit) Template
        vdi-uuid ( RO): 050ed192-8232-412d-aee9-57c34091ae7f
           empty ( RO): false
          device ( RO): xvdd

You are interested in the vdi-uuid value of the device that is xvda.

Now you need to copy the following file in your primary storage to your web server. Make sure your web server can serve .vhd files by adding a mime type of application/vhd


rename it to ubuntu10.04LTSx64.vhd

Removing Templates
xe template-list
xe template-param-set other-config:default_template=false uuid=[uuid]
xe template-param-set is-a-template=false uuid=[uuid]
xe vm-destroy uuid=[uuid]

Testing The Deployment

Host Server (KVM)

On the physical server assume we have Ubuntu 10.04 installed. Configure the network as follows:
apt-get -y update
apt-get -y upgrade
apt-get install -y ntp vim ssh vlan nfs-kernel-server python2.6

sudo modprobe 8021q
sudo sh -c 'grep -q 8021q /etc/modules || echo 8021q >> /etc/modules'

tar -zxvf xenserver-cloud-supp.tgz
xe-install-supplemental-pack xenserver-cloud-supp.iso

Sample /etc/network/interfaces file

auto eth0.116
iface eth0.116 inet static

auto eth1.192
iface eth1.192 inet static

restart the machine and download and install the Cloudstack Agent. This will automatically install all the other packages for you

cd ~
mkdir downloads
cd downloads
mv  CloudStack-oss-3.0.1-1-ubuntu10.04.tar.gz\?r\=http\ CloudStack-oss-3.0.1-1-ubuntu10.04.tar.gz

tar -zxvf CloudStack-oss-3.0.1-1-ubuntu10.04.tar.gz
cd CloudStack-oss-3.0.1-1-ubuntu10.04

Install the Agent, choose “A”

sudo adduser root libvirtd

After the installation, open up the /etc/libvirt/qemu.conf file and uncomment the line below

vnc_listen = ""

Restart libvirtd service

/etc/init.d/libvirt-bin restart

Run the Cloud setup agent


Login to the Management Interface
In this document navigate to

The default username is admin and the password is password

Continue with the Basic Installation

Console and Storage VM’s

Install the Usage Monitor Server (optional)

run the ./ on the server

service cloud-usage start

Provisioning API

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout test-private-key.pem -out test-public-cert.pem

cloud-bridge-register --apikey=Q19kRTgnaHp5JltNSzr71QgpL2qfncQEUmQ3ueDRe63mYitZHmFVjRxsQUoeYl5uVRAIR0qG-yxJEi392xCiPQ --secretkey=r7vr9leCWOh67QYUf9clRIx3RTrKYUM_WXCQRGVrLBJVKBY7nH96KhS_z4LCE_gEMmUDYQJYtc1QpopjHsCWjw --cert=/var/certs/ccslnxubt1110-8639-01-test.pem --url=

tail  /usr/share/cloud/bridge/logs/catalina.out -f

Used for Connected API to  to Cloudstack

Cloudbridge has the following requirements

Cent OS 5 or Higher. In this example we use Cent OS 6.2. This can be installed in a VM.

In this example we have virtualized a Cent 6.2 OS Installation. We are using the Server Install. Download the  CentOS-6.2-x86_64-minimal.iso for this install and also inside a ESXi 5 virtual machine.

Virtual Machine Setup
Install Vmware tools, since you have no network aapter
mount -t iso9660 /dev/cdrom /mnt/cdrom

cd ~
mkdir downloads
cp /mnt/cdrom/VMwareTools-8.6.0-425873.tar.gz downloads
cd downloads
tar -zxvf VMwareTools-8.6.0-425873.tar.gz
cd vmware-tools-distrib/

Download Cloudbridge from Source Forge

cd ~
cd downloads


After the machine is setup run the following commands

setenforce permissive

Install Java and Tomcat

yum install java
cd /etc/yum.repos.d
wget ''
yum install tomcat6 tomcat6-webapps tomcat6-admin-webapps

Install MySQL
yum install mysql mysql-server
service mysqld start

Install Cloudbridge
rpm -ivh cloud-bridge-1.1.0-1.x86_64.rpm

Setup the cloudbridge configuration

service cloud-bridge start

Debugging Cloudbridge

tail /usr/share/cloud/bridge/logs/cloudbridge.log -f
Adding a User to Cloudbridge

Each CloudBridge user must perform a one-time registration. The user follows these steps:

1. Obtain the following from your CloudStack cloud administrator:
The CloudBridge server's publicly available DNS name or IP address
Your account's API key and Secret key - you can generate the keys in the web interface when you create the accounts, then drill down into the view users per account

2. Generate a private key and a self-signed X.509 certificate. Substitute your own desired storage location for /path/to/… below. $ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /path/to/private_key.pem -out /path/to/cert.pem

3. Register the mapping from the X.509 certificate to the API/Secret keys in CloudBridge. Substitute the values you obtained from the CloudStack administrator in the URL below. $ cloud-bridge-register --apikey=<User’s Cloudstack API key> --secretkey=<User’s CloudStack Secret key> --cert=</path/to/cert.pem> --url=http://<cloud-bridge-server>:8090/bridge

Cloudbridge Commands
ec2-describe-availability-zones --connection-timeout 30 --request-timeout 30

ec2-describe-images --connection-timeout 30 --request-timeout 30

ec2-describe-instances --connection-timeout 30 --request-timeout 30

ec2-run-instances 66753dbb-b1d6-49c7-a9d3-e68661e77072 --availability-zone ccs-cs-ca-west-01 --instance-count 2 --instance-type m1.small

Load Balancer

Citrix Netscaler Appliance

Useful commands

Restart the Cloud Management Server
service cloud-management restart

Special Notes

Overprovisioning CPU

By Default, Cloudstack will allocate whatever CPU and cores you have to a certain pool. Under Global Settings the default

cpu.overprovisioning.factor = 1

Which means there is a one to one mapping. Depending on your load, you can choose to overprovision. There is no recommended amount for over provisioning, as it really depends on your workload. If you have very consistent workloads, you will want to keep this value close to 1. However if you have inconsistent workloads and virtual machines tend to stay idle, you can increase this to any number you like, we use the number anywhere between 2 and 6 (4 is the default we have picked.)

Enabling Local Storage
This setting, which is often the first seen by admins tells CloudStack that system VMs (Secondary Storage and Console Proxy) can be created on local storage. = true
This setting, which controls the non-system-VMs enables whether local storage is permitted at all within CloudStack. Note that even after enabling this setting that VMs won't magically be created utilizing local storage, the service offering has to specify local storage. By default = false.

There are a number of situations where shared storage isn't appropriate or at least not optimal. For instance, situations where high availability isn't needed, or where greater disk I/O is needed, using direct attached storage on the hypervisor itself makes a lot of sense.  = true

Deleting and Expunging

Set the clean up intervals and expunge intervals to clear out the virtual machines on the global settings


Ahmad Emneina said...

What an awesome resource!!! thanks for posting this.

Harry Yeh said...

Thanks, too bad there were so many issues with it, I'll be posting something for the newer versions later this year