Veewee, Vagrant and some CloudStack [feedly]

Veewee, Vagrant and some CloudStack
http://feedly.com/e/ng2tW6fE

Coming back from CloudStack conference the feeling that this is not about building clouds got stronger. This is really about what to do with them and how they bring you agility, faster-time to market and allow you to focus on innovation in your core business. A large component of this is Culture and a change of how we do IT. The DevOps movement is the embodiment of this change. Over in Amsterdam I was stoked to meet with folks that I had seen at other locations throughout Europe in the last 18 months. Folks from PaddyPower, SchubergPhilis, Inuits who all embrace DevOps. I also met new folks, including Hugo Correia from Klarna(CloudStack users) who came by to talk about Vagrant-cloudstack plugin. His talk and a demo by Roland Kuipers from Schuberg was enough to kick my butt and get me to finally check out Vagrant. I sprinkled a bit of Veewee and of course some CloudStack on top of it all. Have fun reading.

Automation is key to a reproducible, failure-tolerant infrastructure. Cloud administrators should aim to automate all steps of building their infrastructure and be able to re-provision everything with a single click. This is possible through a combination of configuration management, monitoring and provisioning tools. To get started in created appliances that will be automatically configured and provisioned, two tools stand out in the arsenal: Veewee and Vagrant.

Veewee: Veewee is a tool to easily create appliances for different hypervisors. It fetches the .iso of the distribution you want and build the machine with a kickstart file. It integrates with providers like VirtualBox so that you can build these appliances on your local machine. It supports most commonly used OS templates. Coupled with virtual box it allows admins and devs to create reproducible base appliances. Getting started with veewee is a 10 minutes exericse. The README is great and there is also a very nice post that guides you through your first box building.

Most folks will have no issues cloning Veewee from github and building it, you will need ruby 1.9.2 or above. You can get it via `rvm` or your favorite ruby version manager.

git clone https://github.com/jedi4ever/veewee  gem install bundler  bundle install  

Setting up an alias is handy at this point `alias veewee="bundle exec veewee"`. You will need a virtual machine provider (e.g VirtualBox, VMware Fusion, Parallels, KVM). I personnaly use VirtualBox but pick one and install it if you don't have it already. You will then be able to start using `veewee` on your local machine. Check the sub-commands available (for virtualbox):

$ veewee vbox  Commands:    veewee vbox build [BOX_NAME]                     # Build box    veewee vbox copy [BOX_NAME] [SRC] [DST]          # Copy a file to the VM    veewee vbox define [BOX_NAME] [TEMPLATE]         # Define a new basebox starting from a template    veewee vbox destroy [BOX_NAME]                   # Destroys the virtualmachine that was built    veewee vbox export [BOX_NAME]                    # Exports the basebox to the vagrant format    veewee vbox halt [BOX_NAME]                      # Activates a shutdown the virtualmachine    veewee vbox help [COMMAND]                       # Describe subcommands or one specific subcommand    veewee vbox list                                 # Lists all defined boxes    veewee vbox ostypes                              # List the available Operating System types    veewee vbox screenshot [BOX_NAME] [PNGFILENAME]  # Takes a screenshot of the box    veewee vbox sendkeys [BOX_NAME] [SEQUENCE]       # Sends the key sequence (comma separated) to the box. E.g for testing the :boot_cmd_sequence    veewee vbox ssh [BOX_NAME] [COMMAND]             # SSH to box    veewee vbox templates                            # List the currently available templates    veewee vbox undefine [BOX_NAME]                  # Removes the definition of a basebox     veewee vbox up [BOX_NAME]                        # Starts a Box    veewee vbox validate [BOX_NAME]                  # Validates a box against vagrant compliancy rules    veewee vbox winrm [BOX_NAME] [COMMAND]           # Execute command via winrm    Options:            [--debug]           # enable debugging    -w, --workdir, [--cwd=CWD]  # Change the working directory. (The folder containing the definitions folder).                                # Default: /Users/sebgoa/Documents/gitforks/veewee  

Pick a template from the `templates` directory and `define` your first box:

  veewee vbox define myfirstbox CentOS-6.5-x86_64-minimal  

You should see that a `defintions/` directory has been created, browse to it and inspect the `definition.rb` file. You might want to comment out some lines, like removing `chef` or `puppet`. If you don't change anything and build the box, you will then be able to `validate` the box with `veewee vbox validate myfirstbox`. To build the box simply do:

veewee vbox build myfristbox  

Everything should be successfull, and you should see a running VM in your virtual box UI. To export it for use with `Vagrant`, `veewee` provides an export mechanism (really a VBoxManage command): `veewee vbox export myfirstbox`. At the end of the export, a .box file should be present in your directory.

Vagrant: Picking up from where we left with `veewee`, we can now add the box to Vagrantand customize it with shell scripts or much better, with Puppet recipes or Chef cookbooks. First let's add the box file to Vagrant:

vagrant box add 'myfirstbox' '/path/to/box/myfirstbox.box'  

Then in a directory of your choice, create the Vagrant "project":

     vagrant init 'myfirstbox'  

This will create a `Vagrantfile` that we will later edit to customize the box. You can boot the machine with `vagrant up` and once it's up , you can ssh to it with `vagrant ssh`.

While `veewee` is used to create a base box with almost no customization (except potentially a chef and/or puppet client), `vagrant` is used to customize the box using the Vagrantfile. For example, to customize the `myfirstbox` that we just built, set the memory to 2 GB, add a host-only interface with IP 192.168.56.10, use the apache2 Chef cookbook and finally run a `boostrap.sh` script, we will have the following `Vagrantfile`:

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|      # Every Vagrant virtual environment requires a box to build off of.    config.vm.box = "myfirstbox"    config.vm.provider "virtualbox" do |vb|      vb.customize ["modifyvm", :id, "--memory", 2048]    end      #host-only network setup    config.vm.network "private_network", ip: "192.168.56.10"      # Chef solo provisioning    config.vm.provision "chef_solo" do |chef|       chef.add_recipe "apache2"    end      #Test script to install CloudStack    #config.vm.provision :shell, :path => "bootstrap.sh"      end  

The cookbook will be in a `cookbooks` directory and the boostrap script will be in the root directory of this vagrant definition. For more information, check the Vagrant website and experiment.

Vagrant and CloudStack: What is very interesting with Vagrant is that you can use various plugins to deploy machines on public clouds. There is a `vagrant-aws` plugin and of course a `vagrant-cloudstack` plugin. You can get the latest CloudStack plugin from github. You can install it directly with the `vagrant` command line:

vagrant plugin install vagrant-cloudstack  

Or if you are building it from source, clone the git repository, build the gem and install it in `vagrant`

  git clone https://github.com/klarna/vagrant-cloudstack.git  gem build vagrant-cloudstack.gemspec  gem install vagrant-cloudstack-0.1.0.gem  vagrant plugin install /Users/sebgoa/Documents/gitforks/vagrant-cloudstack/vagrant-cloudstack-0.1.0.gem  

The only drawback that I see is that one would want to upload his local box (created from the previous section) and use it. Instead one has to create `dummy boxes` that use existing templates available on the public cloud. This is easy to do, but creates a gap between local testing and production deployments. To build a dummy box simply create a `Vagrantfile` file and a `metadata.json` file like so:

$ cat metadata.json   {      "provider": "cloudstack"  }  $ cat Vagrantfile   # -*- mode: ruby -*-  # vi: set ft=ruby :    Vagrant.configure("2") do |config|    config.vm.provider :cloudstack do |cs|      cs.template_id = "a17b40d6-83e4-4f2a-9ef0-dce6af575789"    end  end  

Where the `cs.template_id` is a uuid of a CloudStack template in your cloud. CloudStack users will know how to easily get those uuids with `CloudMonkey`. Then create a `box` file with `tar cvzf cloudstack.box ./metadata.json ./Vagrantfile`. Note that you can add additional CloudStack parameters in this box definition like the host,path etc (something to think about :) ). Then simply add the box in `Vagrant` with:

vagrant box add ./cloudstack.box  

You can now create a new `Vagrant` project:

  mkdir cloudtest  cd cloudtest  vagrant init  

And edit the newly created `Vagrantfile` to use the `cloudstack` box. Add additional parameters like `ssh` configuration, if the box does not use the default from `Vagrant`, plus `service_offering_id` etc. Remember to use your own api and secret keys and change the name of the box to what you created. For example on exoscale:

# -*- mode: ruby -*-  # vi: set ft=ruby :    # Vagrantfile API/syntax version. Don't touch unless you know what you're doing!  VAGRANTFILE_API_VERSION = "2"    Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|      # Every Vagrant virtual environment requires a box to build off of.    config.vm.box = "cloudstack"      config.vm.provider :cloudstack do |cs, override|      cs.host = "api.exoscale.ch"      cs.path = "/compute"      cs.scheme = "https"      cs.api_key = "PQogHs2sk_3..."      cs.secret_key = "...NNRC5NR5cUjEg"      cs.network_type = "Basic"        cs.keypair = "exoscale"      cs.service_offering_id = "71004023-bb72-4a97-b1e9-bc66dfce9470"      cs.zone_id = "1128bd56-b4d9-4ac6-a7b9-c715b187ce11"        override.ssh.username = "root"       override.ssh.private_key_path = "/path/to/private/key/id_rsa_example"    end      # Test bootstrap script    config.vm.provision :shell, :path => "bootstrap.sh"    end  

The machine is brought up with:

vagrant up --provider=cloudstack  

The following example output will follow:

  $ vagrant up --provider=cloudstack  Bringing machine 'default' up with 'cloudstack' provider...  [default] Warning! The Cloudstack provider doesn't support any of the Vagrant  high-level network configurations (`config.vm.network`). They  will be silently ignored.  [default] Launching an instance with the following settings...  [default]  -- Service offering UUID: 71004023-bb72-4a97-b1e9-bc66dfce9470  [default]  -- Template UUID: a17b40d6-83e4-4f2a-9ef0-dce6af575789  [default]  -- Zone UUID: 1128bd56-b4d9-4ac6-a7b9-c715b187ce11  [default]  -- Keypair: exoscale  [default] Waiting for instance to become "ready"...  [default] Waiting for SSH to become available...  [default] Machine is booted and ready for use!  [default] Rsyncing folder: /Users/sebgoa/Documents/exovagrant/ => /vagrant  [default] Running provisioner: shell...  [default] Running: /var/folders/76/sx82k6cd6cxbp7_djngd17f80000gn/T/vagrant-shell20131203-21441-1ipxq9e  Tue Dec  3 14:25:49 CET 2013  This works  

Which is a perfect execution of my amazing bootstrap script:

  #!/usr/bin/env bash    /bin/date  echo "This works"  

You can now start playing with Chef cookbooks, Puppet recipes or SaltStack formulas and automate the configuration of your cloud instances, thanks to Veewee Vagrant andCloudStack.