Thursday, June 07, 2012

Cloudstack Infrastructure Build and Deployment - Ubuntu 10.04


* This is one of the docs I wrote about deploying Cloudstack Build and Deployment on Ubuntu 10.04. There were some issues running it properly with Xen Hypervisor so I never fully completed or published this. In the meantime, Ubuntu 12.04 also came out and there's almost nothing to date with Cloudstack being updated with to work with 12.04. I will probably continue this at a later date.
Management Server
This install assumes the use of Ubuntu 10.04 and assumes you are logged in as root.

The Management Server in this example is installed on a vmware machine with 1 x 120 gig disk and 1 x 500 gig disk.

apt-get -y update
apt-get -y upgrade
apt-get install -y ntp nfs-kernel-server
mkdir /etc/exports.d




Also in this example we have a 1.5TB second drive setup at /dev/cciss/c1d0p1 do the following

fdisk /dev/cciss/c1d0
n
p
enter
t
l
8e
w
pvcreate /dev/cciss/c1d0p1
pvdisplay

mkfs.ext4 /dev/cciss/c1d0p1


vim /etc/fstab


Add an entry to the bottom of /etc/fstab then restart the server

/dev/cciss/c1d0p1       /srv        ext4    defaults,auto,_netdev 0 0

Disable IPV6
vim /etc/sysctl.conf
Add the following line to the end of the file /etc/sysctl.conf (as root or use sudo)
net.ipv6.conf.all.disable_ipv6 = 1
Run the following command to make this change effective immediately

Restart the Computer


cd ~
mkdir downloads
cd downloads
wget http://downloads.sourceforge.net/project/cloudstack/CloudStack%20Acton/3.0.1/CloudStack-oss-3.0.1-1-ubuntu10.04.tar.gz
tar -zxvf CloudStack-oss-3.0.1-1-ubuntu10.04.tar.gz
cd CloudStack-oss-3.0.1-1-ubuntu10.04

Updating
wget http://downloads.sourceforge.net/project/cloudstack/CloudStack%20Acton/3.0.2/CloudStack-oss-3.0.2-1-ubuntu10.04.tar.gz
tar -zxvf CloudStack-oss-3.0.2-1-ubuntu10.04.tar.gz
cd CloudStack-oss-3.0.2-1-ubuntu10.04

./install.sh
(Choose the Upgrade Option)

mySQL Database
If you didn’t install mySQL, when you run ./install.sh it will prompt you. Select “D”

Assuming mySQL installed open up /etc/mysql/my.cnf and add the following to the
[mysqld] section

innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=350
log-bin=mysql-bin
binlog-format = 'ROW'
#default bind address is 127.0.0.1 change to 0.0.0.0
bind-address = 0.0.0.0

Enable Root access from any machine
mysql -p
grant all on *.* to root@'%' identified by 'password';

Exit and restart mysql

service mysql restart

Install the Management server and choose “M” as the option

./install.sh


Login to mysql and create the cloud user and database

cloud-setup-databases cloud:cloudpassword@localhost --deploy-as=root:password

Choose M for Management Server Software. After the install is complete and done, restart the following services:

Make sure the NFS and RPCBind services are started

service nfs-kernel-server start



Run the command

cloud-setup-management

Prepare the NFS Shares for Storage
mkdir -p /srv/export/primary
mkdir -p /srv/export/secondary

vim /etc/exports

Open up the /etc/exports and add the following

/srv/export *(rw,async,no_root_squash)

vim /etc/idmapd.conf

update the /etc/idmapd.conf

domain = test.com

Show the exports
showmount -e
Download the System VM Templates

/usr/lib/cloud/agent/scripts/storage/secondary/cloud-install-sys-tmplt -m /srv/export/secondary -u http://download.cloud.com/templates/acton/acton-systemvm-02062012.qcow2.bz2 -h kvm -F

/usr/lib/cloud/agent/scripts/storage/secondary/cloud-install-sys-tmplt -m /srv/export/secondary -u http://download.cloud.com/templates/acton/acton-systemvm-02062012.ova -h vmware -F

/usr/lib/cloud/agent/scripts/storage/secondary/cloud-install-sys-tmplt -m /srv/export/secondary -u http://download.cloud.com/templates/acton/acton-systemvm-02062012.vhd.bz2 -h xenserver -F

To Debug or View the system logs go to

/var/log/cloud/management/management-server.log

These logs are very important to review as they will tell you exactly why or why you could not add a host to the cloud infrastructure

on a linux machine or ssh console use the tail command so you can view the log updates

tail /var/log/cloud/management/management-server.log -f
Host Server (Xen)
Xen allows the use of AMI (Amazon Machine Images) for compatibility

XenServer is actually a stand alone server product, for this release download 6.0.2 from. Down the iso, mount and install to a physical server. Be sure not to use VLAN for the management interface as Cloudstack doesn’t support this.

https://www.citrix.com/site/SS/downloads/details.asp?downloadId=2320418&productId=683148


service nfs start
cd ~
mkdir downloads
cd downloads
wget http://download.cloud.com/releases/3.0.1/XS-6.0.2/xenserver-cloud-supp.tgz

tar -zxvf xenserver-cloud-supp.tgz

xe-install-supplemental-pack xenserver-cloud-supp.iso


Advanced Networking
xe-switch-network-backend openvswitch

Basic Networking
xe-switch-network-backend bridge

Then reboot

Test and Launch your instance

After you can launch an instance

default centOS Install
root
password

Importing an Amazon Machine Image

cd ~
cd downloads
mkdir ami
cd ami
wget http://cloud-images.ubuntu.com/releases/oneiric/release/ubuntu-11.10-server-cloudimg-amd64.tar.gz
mkdir -p /mnt/loop/ubuntu1110x64
tar -zxvf ubuntu-11.10-server-cloudimg-amd64.tar.gz
mount -o loop oneiric-server-cloudimg-amd64.img /mnt/loop/ubuntu1110x64/

Creating a new VM Template

You can download this script from http://community.citrix.com/download/attachments/148701904/makeubuntu.sh?version=1 or use the one below. Simply change the distro for newer ubuntu versions

Make sure the file below is executed on the Xenserver host conosle

cd ~
mkdir downloads
cd downloads
wget http://community.citrix.com/download/attachments/148701904/makeubuntu.sh
chmod 700 makeubuntu.sh
./makeubuntu.sh

makeubuntu.sh
#In this example we will create and Ubuntu 10.0.4 LTS Template for XenServer
#!/bin/bash
## makeubuntu.sh, this creates Ubuntu server 10.04 32 and 64 bit templates on Xenserver 5.6. Net install only
## Author: David Markey <david.markey@citrix.com>
## This is not an officially supported guest OS on XenServer 5.6

LENNY=$(xe template-list name-label=Debian\ Lenny\ 5.0\ \(32-bit\) --minimal)

if [[ -z $LENNY ]] ; then
   echo "Cant find lenny 32bit template, is this on 5.6?"
   exit 1
fi

distro="Ubuntu 10.04"
arches=("32-bit" "64-bit")


for arch in ${arches[@]} ; do
   echo "Attempting $distro ($arch)"
   if [[ -n $(xe template-list name-label="$distro ($arch)" params=uuid --minimal) ]] ; then
       echo "$distro ($arch)" already exists, Skipping
   else

       NEWUUID=$(xe vm-clone uuid=$LENNY new-name-label="$distro ($arch)")
       xe template-param-set uuid=$NEWUUID other-config:install-methods=http,ftp \
        other-config:install-repository=http://archive.ubuntu.net/ubuntu \
        PV-args="-- quiet console=hvc0 partman/default_filesystem=ext3" \
        other-config:debian-release=lucid \
        other-config:default_template=true

       if [[ "$arch" == "32-bit" ]] ; then
           xe template-param-set uuid=$NEWUUID other-config:install-arch="i386"
       else
           xe template-param-set uuid=$NEWUUID other-config:install-arch="amd64"
       fi
       echo "Success"
   fi

 

done


echo "Done"
Installing Ubuntu 10.04 LTS



In this example, you will need to create the Ubuntu Virtual Machine Template. The instructions are at http://community.citrix.com/display/xs/Installing+Ubuntu+Server+10.04+%2832bit+and+64bit%29+LTS but I will also let them here

During the Install simply follow the screenshots below

Install XenTools
change to the root user

su
cd ~
mkdir /mnt/cdrom
mount /dev/cdrom /mnt/cdrom
dpkg -i /mnt/cdrom/Linux/xe-guest-utilities_6.0.2-756_amd64.deb

also make sure to set eth0 as DHCP in the /etc/network/interfaces file

auto eth0
iface eth0 inet dhcp

After this is done, shut down the virtual machine and connect to the console of the Xen server where you created the Virtual Machine.

You will also need a web server for this in order to serve the .vhd file to your management server where you are creating the template.

Root into your xen console server
type

xe vm-list

this will show you a list of the virtual machines.

Choose the UUID you just created. this example the command shows this output:


uuid ( RO)           : 2d6b4461-6c11-9491-1c34-b2b9914435ac
    name-label ( RW): Ubuntu 10.04 (64-bit) Template
   power-state ( RO): halted

Now run the command to list the storage associated with the template.

xe vbd-list vm-uuid=2d6b4461-6c11-9491-1c34-b2b9914435ac

This is the output

uuid ( RO)             : b10a26c2-2bca-f2fc-8bca-d8325b5ef873
         vm-uuid ( RO): 2d6b4461-6c11-9491-1c34-b2b9914435ac
   vm-name-label ( RO): Ubuntu 10.04 (64-bit) Template
        vdi-uuid ( RO): 2d4b29a1-07b8-4a26-9dd1-188bed075e0e
           empty ( RO): false
          device ( RO): xvda


uuid ( RO)             : 4ca40894-5464-1f29-e84d-e7ac0f139a8f
         vm-uuid ( RO): 2d6b4461-6c11-9491-1c34-b2b9914435ac
   vm-name-label ( RO): Ubuntu 10.04 (64-bit) Template
        vdi-uuid ( RO): 050ed192-8232-412d-aee9-57c34091ae7f
           empty ( RO): false
          device ( RO): xvdd

You are interested in the vdi-uuid value of the device that is xvda.

Now you need to copy the following file in your primary storage to your web server. Make sure your web server can serve .vhd files by adding a mime type of application/vhd

2d4b29a1-07b8-4a26-9dd1-188bed075e0e.vhd

rename it to ubuntu10.04LTSx64.vhd
 

Removing Templates
xe template-list
xe template-param-set other-config:default_template=false uuid=[uuid]
xe template-param-set is-a-template=false uuid=[uuid]
xe vm-destroy uuid=[uuid]





Testing The Deployment



Host Server (KVM)

On the physical server assume we have Ubuntu 10.04 installed. Configure the network as follows:
apt-get -y update
apt-get -y upgrade
apt-get install -y ntp vim ssh vlan nfs-kernel-server python2.6

sudo modprobe 8021q
sudo sh -c 'grep -q 8021q /etc/modules || echo 8021q >> /etc/modules'

tar -zxvf xenserver-cloud-supp.tgz
xe-install-supplemental-pack xenserver-cloud-supp.iso

Sample /etc/network/interfaces file

auto eth0.116
iface eth0.116 inet static
address 10.100.16.11
netmask 255.255.255.0
gateway 10.100.16.1

auto eth1.192
iface eth1.192 inet static
address 192.168.3.1
netmask 255.255.255.0



restart the machine and download and install the Cloudstack Agent. This will automatically install all the other packages for you

cd ~
mkdir downloads
cd downloads
wget http://downloads.sourceforge.net/project/cloudstack/CloudStack%20Acton/3.0.1/CloudStack-oss-3.0.1-1-ubuntu10.04.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fcloudstack%2Ffiles%2FCloudStack%2520Acton%2F3.0.1%2F
mv  CloudStack-oss-3.0.1-1-ubuntu10.04.tar.gz\?r\=http\:%2F%2Fsourceforge.net%2Fprojects%2Fcloudstack%2Ffiles%2FCloudStack%20Acton%2F3.0.1%2F CloudStack-oss-3.0.1-1-ubuntu10.04.tar.gz

tar -zxvf CloudStack-oss-3.0.1-1-ubuntu10.04.tar.gz
cd CloudStack-oss-3.0.1-1-ubuntu10.04
./install.sh



Install the Agent, choose “A”

sudo adduser root libvirtd

After the installation, open up the /etc/libvirt/qemu.conf file and uncomment the line below

vnc_listen = "0.0.0.0"

Restart libvirtd service

/etc/init.d/libvirt-bin restart

Run the Cloud setup agent

cloud-setup-agent


Login to the Management Interface
In this document navigate to http://10.100.16.10:8080/client

The default username is admin and the password is password

Continue with the Basic Installation

Console and Storage VM’s

root
6m1ll10n
Install the Usage Monitor Server (optional)

run the ./install.sh on the server

service cloud-usage start


Provisioning API

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout test-private-key.pem -out test-public-cert.pem



cloud-bridge-register --apikey=Q19kRTgnaHp5JltNSzr71QgpL2qfncQEUmQ3ueDRe63mYitZHmFVjRxsQUoeYl5uVRAIR0qG-yxJEi392xCiPQ --secretkey=r7vr9leCWOh67QYUf9clRIx3RTrKYUM_WXCQRGVrLBJVKBY7nH96KhS_z4LCE_gEMmUDYQJYtc1QpopjHsCWjw --cert=/var/certs/ccslnxubt1110-8639-01-test.pem --url=http://ccs-cs-bridge.test.com:8090/bridge

Debugging
tail  /usr/share/cloud/bridge/logs/catalina.out -f


Cloudbridge
Used for Connected API to  to Cloudstack

Cloudbridge has the following requirements

Cent OS 5 or Higher. In this example we use Cent OS 6.2. This can be installed in a VM.

In this example we have virtualized a Cent 6.2 OS Installation. We are using the Server Install. Download the  CentOS-6.2-x86_64-minimal.iso for this install and also inside a ESXi 5 virtual machine.

Virtual Machine Setup
Install Vmware tools, since you have no network aapter
mount -t iso9660 /dev/cdrom /mnt/cdrom

cd ~
mkdir downloads
cp /mnt/cdrom/VMwareTools-8.6.0-425873.tar.gz downloads
cd downloads
tar -zxvf VMwareTools-8.6.0-425873.tar.gz
cd vmware-tools-distrib/
./vmware-install.pl

Download Cloudbridge from Source Forge
http://sourceforge.net/projects/cloudstack/files/CloudBridge/




cd ~
cd downloads

wget http://downloads.sourceforge.net/project/cloudstack/CloudBridge/cloud-bridge-1.1.0-1.x86_64.rpm?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fcloudstack%2Ffiles%2FCloudBridge%2F&ts=1337076256&use_mirror=voxel


After the machine is setup run the following commands


setenforce permissive

Install Java and Tomcat

yum install java
cd /etc/yum.repos.d
wget 'http://jpackage.org/jpackage50.repo'
yum install tomcat6 tomcat6-webapps tomcat6-admin-webapps

Install MySQL
yum install mysql mysql-server
service mysqld start

Install Cloudbridge
rpm -ivh cloud-bridge-1.1.0-1.x86_64.rpm

Setup the cloudbridge configuration
cloud-setup-bridge

cloud-setup-bridge-db
service cloud-bridge start

Debugging Cloudbridge

tail /usr/share/cloud/bridge/logs/cloudbridge.log -f
Adding a User to Cloudbridge

Each CloudBridge user must perform a one-time registration. The user follows these steps:

1. Obtain the following from your CloudStack cloud administrator:
The CloudBridge server's publicly available DNS name or IP address
Your account's API key and Secret key - you can generate the keys in the web interface when you create the accounts, then drill down into the view users per account

2. Generate a private key and a self-signed X.509 certificate. Substitute your own desired storage location for /path/to/… below. $ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /path/to/private_key.pem -out /path/to/cert.pem

3. Register the mapping from the X.509 certificate to the API/Secret keys in CloudBridge. Substitute the values you obtained from the CloudStack administrator in the URL below. $ cloud-bridge-register --apikey=<User’s Cloudstack API key> --secretkey=<User’s CloudStack Secret key> --cert=</path/to/cert.pem> --url=http://<cloud-bridge-server>:8090/bridge

Cloudbridge Commands
ec2-describe-availability-zones
ec2-describe-availability-zones --connection-timeout 30 --request-timeout 30

ec2-describe-images --connection-timeout 30 --request-timeout 30

ec2-describe-instances --connection-timeout 30 --request-timeout 30


ec2-run-instances 66753dbb-b1d6-49c7-a9d3-e68661e77072 --availability-zone ccs-cs-ca-west-01 --instance-count 2 --instance-type m1.small


Load Balancer

Citrix Netscaler Appliance



Useful commands

Restart the Cloud Management Server
service cloud-management restart


Special Notes

Overprovisioning CPU

By Default, Cloudstack will allocate whatever CPU and cores you have to a certain pool. Under Global Settings the default

cpu.overprovisioning.factor = 1

Which means there is a one to one mapping. Depending on your load, you can choose to overprovision. There is no recommended amount for over provisioning, as it really depends on your workload. If you have very consistent workloads, you will want to keep this value close to 1. However if you have inconsistent workloads and virtual machines tend to stay idle, you can increase this to any number you like, we use the number anywhere between 2 and 6 (4 is the default we have picked.)

Enabling Local Storage

system.vm.use.local.storage:
This setting, which is often the first seen by admins tells CloudStack that system VMs (Secondary Storage and Console Proxy) can be created on local storage.

system.vm.use.local.storage = true

use.local.storage:
This setting, which controls the non-system-VMs enables whether local storage is permitted at all within CloudStack. Note that even after enabling this setting that VMs won't magically be created utilizing local storage, the service offering has to specify local storage. By default use.local.storage = false.

There are a number of situations where shared storage isn't appropriate or at least not optimal. For instance, situations where high availability isn't needed, or where greater disk I/O is needed, using direct attached storage on the hypervisor itself makes a lot of sense.

use.local.storage  = true



Deleting and Expunging

Set the clean up intervals and expunge intervals to clear out the virtual machines on the global settings

2 comments:

Ahmad Emneina said...

What an awesome resource!!! thanks for posting this.

Harry Yeh said...

Thanks, too bad there were so many issues with it, I'll be posting something for the newer versions later this year