Once upon a time, in a time far, far away (don't most good stories start this way?) XenServer was so easy to get installed and running that we promoted it as "Ten Minutes to Xen". While this is still often the case for small installations, even ten minutes can be problematic for some, and even more so when hundreds of hosts are involved. In this article, we'll expand upon the XenServer Quick Installation Guide and show how you can scale out your XenServer environment quickly using a scripting model, and ensure you have correct monitoring and logging in place by default.
Assumptions
This article assumes you've already installed XenServer on one server and validated that no additional drivers are required. It also assumes that you've configured you server BIOS to be identical across all servers, and that PXE is supported on the NIC used as the management network. One key item in the preparation is that the servers are set to boot in legacy BIOS mode and not UEFI.
Preparation steps
1. Download the XenServer installation ISO media: http://xenserver.org/open-source-virtualization-download.html
2. Extract the entire contents of XenServer installation ISO file to either a HTTP, FTP or NFS location (in this example we'll be using NFS)
3. Collect the following information
Hostname: xenserver
Root password: password
Keyboard locale: us
NTP server address: 0.us.pool.ntp.org
DNS server address: dns.local
Time zone: America/New_York (supported time zones in RHEL)
Location of extracted ISO file: nfsserver:/
TFTP server IP address: pxehost
Configuring TFTP server to supply XenServer installer
1. In the /tftpboot directory create a new directory called xenserver
2. Copy the mboot.c32 and pxelinux.0 files from the /boot/pxelinux directory of the XenServer ISO file to the /tftpboot directory
3. Copy the install.img file from the root directory of the XenServer ISO file to the /tftpboot/xenserver directory
4. Copy the vmlinuz and zen.gz files from the /boot directory of the XenServer ISO file to the /tftpboot/xenserver directory
5. In the /tftpboot directory, create a new directory called pxelinux.cfg
The above steps are covered in this script:
mkdir /mnt/xsinstall mount [XenServer ISO Extract Location] /mnt/xsinstall cd ./tftpboot mkdir xenserver cp /mnt/xsinstall/boot/pxelinux/mboot.c32 ./ cp /mnt/xsinstall/boot/pxelinux/pxelinux.0 ./ cp /mnt/xsinstall/install.img ./xenserver cp /mnt/xsinstall/boot/vmlinuz ./xenserver cp /mnt/xsinstall/boot/zen.gz./xenserver 6. In the /tftpboot/pxelinux.cfg directory create a new configuration file called default
7. Edit the default file to contain the following information. Note that this command includes remote logging to a SYSLOG server.
default xenserver-auto label xenserver-auto kernel mboot.c32 append xenserver/xen.gz dom0_max_vcpus=1-2 dom0_mem=752M,max:752M com1=115200,8n1 console=com1,vga --- xenserver/vmlinuz xencons=hvc console=hvc0 console=tty0 answerfile=http://[pxehost]/answerfile.xml remotelog=[SYSLOG] install --- xenserver/install.img 8. Unattended installation of XenServer requires an answer file. Place the answer file in the root directory of your NFS server. Please note that there are many more options than are listed here, but this will suffice for most installations.
<?xml version="1.0"?> <installation mode="fresh" srtype="lvm"> <bootloader>extlinux</bootloader> <primary-disk gueststorage="yes">sda</primary-disk> <keymap>[keyboardmap]</keymap> <hostname>[hostname]</hostname> <root-password>[password]</root-password> <source type="nfs">[XenServer ISO Extract Location]</source> <admin-interface name="eth0" proto="dhcp"/> <name-server>dns.local</name-server> <timezone>[Time zone]</timezone> <time-config-method>ntp</time-config-method> <ntp-server>[NTP Server Address]</ntp-server> <script stage="filesystem-populated" type="nfs">[XenServer ISO Extract Location]/post-install-script.sh</script> </installation>
Configuring the post installation scripts
1. In the root directory of the XenServer ISO extract location, create a file named post-install-script.sh with the following contents. This script will run after a successful installation, and copies a first boot script for post installation configuration.
#!/bin/sh touch $1/tmp/post-executed mkdir $1/mnt/xsinstall mount [XenServer ISO Extract Location] $1/mnt/xsinstall cp $1/mnt/xsinstall/first-boot-script.sh $1/var/xen/fbs.sh chmod 777 $1/var/xen/fbs.sh ln -s /var/xen/fbs.sh $1/etc/rc3.d/S99zzzzpostinstall
2. In the root directory of the XenServer ISO extract location, create a file named first-boot-script.sh with whatever steps you need to configure XenServer for your environment. In the script below, we take care of the following cases:
a. Assign a unique, human understandable hostname based on the assigned IP address
b. Configure a dedicated storage network which uses Jumbo frames
c. Configure centralized logging using SYSLOG
d. Configure network monitoring using NetFlow
e. Apply a socket based license
f. Remove the first script to ensure it doesn't run on subsequent reboots
#!/bin/bash # Wait before start sleep 60 # Get current hostname which then gets us the host-uuid HOSTNAME=$(hostname) HOSTUUID=$(xe host-list name-label=$HOSTNAME --minimal) # Get the management pif UUID which gets us the IP address MGMTPIFUUID=$(xe pif-list params=uuid management=true host-name-label=$HOSTNAME --minimal) MGMTIP=$(xe pif-param-list uuid=$MGMTPIFUUID | grep 'IP '| sed -n 's/.*: ([0-9.]*)/1/p') # From the IP address, get the zone and host ZONE=$(echo "$MGMTIP" | awk -F: '{ split($1,a,"."); printf ("%dn", a[3]); }') HOST=$(echo "$MGMTIP" | awk -F: '{ split($1,a,"."); printf ("%dn", a[4]); }') # Configure SYSLOG xe host-param-set uuid=$HOSTUUID logging:syslog_destination=[SYSLOG] xe host-syslog-reconfigure host-uuid=$HOSTUUID # Assign License to server xe host-apply-edition edition=per-socket host-uuid=$HOSTUUID license-server-address=[LicenseServer] license-server-port=27000 # Setup storage network. For us, that's on eth1 (aka xenbr1) STORAGEPIFUUID=$(xe pif-list params=uuid host-name-label=$HOSTNAME device=eth1 --minimal) xe pif-reconfigure-ip mode=static uuid=$STORAGEPIFUUID ip=192.168.$ZONE.$HOST netmask=255.255.255.0 xe pif-param-set disallow-unplug=true uuid=$STORAGEPIFUUID xe pif-param-set other-config:management_purpose="Storage" uuid=$STORAGEPIFUUID NETWORKUUID=$(xe network-list params=uuid bridge=xenbr1 –minimal) xe network-param-set uuid=$NETWORKUUID MTU=9000 # Setup NetFlow monitoring on the 4 network bridges in our hosts ovs-vsctl set Bridge xenbr0 netflow=@nf -- --id=@nf create NetFlow targets="192.168.0.34:5566" active-timeout=30 ovs-vsctl set Bridge xenbr1 netflow=@nf -- --id=@nf create NetFlow targets="192.168.0.34:5566" active-timeout=30 ovs-vsctl set Bridge xenbr2 netflow=@nf -- --id=@nf create NetFlow targets="192.168.0.34:5566" active-timeout=30 ovs-vsctl set Bridge xenbr3 netflow=@nf -- --id=@nf create NetFlow targets="192.168.0.34:5566" active-timeout=30 # Rename host in both XenServer and for XenCenter NEWHOSTNAME=$(echo $HOSTNAME$ZONE-$HOST) xe host-set-hostname-live host-uuid=$HOSTUUID host-name="$NEWHOSTNAME " xe host-param-set uuid=$HOSTUUID name-label="$NEWHOSTNAME"
# Disable first boot script for subsequent reboots rm -f /etc/rc3.d/S99zzzzpostinstall # Final Reboot reboot
Configuring the network
There are several considerations we need to account for in our network design.
1. The XenServer management networks cannot be tagged within XenServer. To work around this, the network ports will need to have a default VLAN assigned to them.
2. The storage management network is using jumbo frames and will need an MTU of 9000
3. The TFTP server will need to be on the primary management network
4. Since we will want to have persistent control over the XenServer hosts and their VMs, we will want to have each XenServer use a static address. In order to accomplish with DHCP, we'll need to configure our DHCP service to use static MAC address reservations. A sample dhcpd.conf is provided below:
authoritative; dns-update-style interim; default-lease-time 28800; max-lease-time 28800; option routers 10.10.2.1; option broadcast-address 10.10.2.255; option subnet-mask 255.255.255.0; option domain-name-servers 10.10.2.2, 10.10.2.3; option domain-name "xspool.local"; subnet 10.10.2.0 netmask 255.255.255.0 { pool { range 10.10.2.50 10.10.2.250; # one host entry following our naming convention host xenserver2-50 { hardware ethernet 00:11:22:33:44:55; fixed-address 10.10.2.50; } host xenserver2-51 { hardware ethernet 00:11:22:33:44:56; fixed-address 10.10.2.51; } host xenserver2-52 { hardware ethernet 00:11:22:33:44:57; fixed-address 10.10.2.52; } # prevent unknown hosts from polluting the pool deny unknown-clients; }
Booting the servers to perform the install
Since our objective is to perform a scale installation using scripting, we also need to script the PXE boot of our servers, and ensure the PXE boot is a first boot only (i.e. we're not continuously reinstalling on each reboot). Thankfully remote access cards provide this capability, and I'm currently compiling a set of scripts to cover as many vendors as I can.
Tying it all together
In this article you've seen how easy it is to deploy a large number of XenServer hosts consistently. That's not the end of things, and over the coming weeks I'll be posting guides covering many more scale operations with XenServer.
No comments:
Post a Comment