Visualizing the CloudStack Planners in Action [feedly]

Visualizing the CloudStack Planners in Action
http://feedproxy.google.com/~r/posterous/wbyo/~3/--ggiIujzVI/visualizing-the-cloudstack-planners-in-action.html

I've started to expore ways to use the CloudStack simulator to help provide more accurate capacity modeling information, in order to help users understand how user behaviour and environment design / growth patterns will effect an environment's capacity.  This is an early step along that path, but it was enough of a starting point that I figured I'd share.

Visualizations

There are 2 visualizations below.  Both are generated through a rather messy process of simulating load against cloudstack, collecting data from the db at 5 second intervals, processing that data into json objects and finally putting them together using the cool svg functions of the d3.js javascript library.

I'll walk through the process below, but here are two planners in action:

How to do this?

Steps to reproduce (assumes you have a local management server and mysql database):

1. Test Script to Drive the sScenario: Copy the contents of this gist into a file named test_deploy_lots_of_vms_with_firstfitplanner.py within the test/integration/smoke/ folder.
2. Simulator Configuration: Copy the contents of this gist into a file named advanced-32host.cfg within the setup/dev folder.
3. Compile and Configure: run the following: mvn -Pdeveloper -Dsimulator clean install; mvn -Pdeveloper -pl developer -Ddeploydb; mvn -Pdeveloper -pl developer -Ddeploydb-simulator; mvn -pl client jetty:run
4. Setup Zone: Once the mgmt server is running, in a new terminal run the followign: mvn -Pdeveloper,marvin.setup -Dmarvin.config=setup/dev/advanced-32host.cfg -pl :cloud-marvin integration-test
5. Restart Mgmt Server: In order for the new global settings to take effect, CTRL-C on the mgmt server's terminal, wait for it to start, and then run the following to restart it: mvn -pl client jetty:run
6. Capture Host Data: In a new terminal window, run the following to collect stats:  rm testout.txt; while true; do `sudo mysql cloud -e "select pod_name, cluster_name, name, cpus, speed, ram, memory_used_capacity, cpu_used_capacity from host_view;" >> testout.txt`; sleep 5; done
7. Run the Scenario: In yet another terminal window, run: nosetests --with-marvin --marvin-config=setup/dev/advanced-32host.cfg --load test/integration/smoke/test_deploy_lots_of_vms_with_firstfitplanner.py

At this point, you should have the processes collecting data and the tests running to populate the simulated zone.  Once the "nosetest" command completes, cancel the mysql collection loop as well.

In order to process the mysql output, I created this simple script.  Download it, chmod u+x and then execute it (change the file names where needed).  The result will be a javascript file that you can load into a page similar to this one.

What's Next to Try?

This is quite rudamentary, but I'm going to work on this a bit to enable a richer set of simulation options, including multiple accounts, varied VM size requests, VM deletions, etc...  The other big component that I believe is interesting is setting a capacity augmentation trigger point, along with some assumption for how much capacity is added to the zone at any one time.

Let me know if you found this interesting...  and if there are any areas that you think are worth exploring.