Recovering From a vCenter Failure [feedly]

----
Recovering From a vCenter Failure
// CloudStack Consultancy & CloudStack...

While, in my opinion VMware's vSphere is the best performing and most stable hypervisor available, vCenter obstinately remains a single point of failure when using vSphere and it's no different when leveraging vCenter in a CloudStack environment.  Therefore, very occasionally there is a requirement to rebuild a vCentre server which was previously running in your CloudStack environment.  Working with one of our clients we found out (the hard way) how to do this.

Replacing a Pre-Existing vCenter Instance

Recreating a vCenter Instance

The recovery steps below apply to both the Windows-based vCenter server or an appliance.

The first step, which I won't cover in detail here, is to build a new vCenter server and attach the hosts to it.  Follow VMware standard procedure when building your new vCenter server.  To save some additional steps later on, reuse the host name and IP address of the previous vCenter server. Ensure that the permissions have been reapplied allowing CloudStack to connect at the vCentre level with full administrative privileges and that datacenter and cluster names have been recreated accurately.  When you re-add the hosts, the VMs running on those hosts will automatically be pulled into the vCentre inventory.

Networking

The environment that we were working on was using standard vSwitches; therefore the configuration of those switches was held on each of the hosts independently and pulled into the vCenter inventory when the hosts were re-added to vCenter.  Distributed vSwitches (dvSwitches) have their configuration held in the vCenter.  We did not have to deal with this complication and so the recreation of dvSwitches is not covered here.

Roll-back

The changes that are to be made can easily be undone, as they make no change and trigger no change in the physical environment.  However, it is never a bad idea to have a backup, you shouldn't ever need to roll back the whole database, but it's good to have a record of your before/after states.

Update vCenter Password in CloudStack

If you have changed the vCenter password, it'll need updating in the CloudStack database in its encrypted form.  The following instructions are taken from the CloudStack documentation:

• Generate the encrypted equivalent of your vCenter password:

$ java -classpath /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar org.jasypt.intf.cli.JasyptPBEStringEncryptionCLI encrypt.sh input="_your_vCenter_password_" password="`cat /etc/cloudstack/management/key`" verbose=false

• Store the output from this step, we need to add this in cluster_details table and vmware_data_center tables in place of the plain text password
• Find the ID of the row of cluster_details table that you have to update:

$ mysql -u <USERNAME> -p<PASSWORD>  select * from cloud.cluster_details;

• Update the password with the new encrypted one

update cloud.cluster_details set value = '_ciphertext_from_step_1_' where id = _id_from_step_2_;

• Confirm that the table is updated:

select * from cloud.cluster_details;

• Find the ID of the correct row of vmware_data_center that you want to update

select * from cloud.vmware_data_center;

• update the plain text password with the encrypted one:

update cloud.vmware_data_center set password = '_ciphertext_from_step_1_' where id = _id_from_step_5_;

• Confirm that the table is updated:

select * from cloud.vmware_data_center;

Reallocating the VMware Datacenters to CloudStack

The next step is to recreate the cloud.zone property within vSphere which is used to track VMware datacenters which have been connected to CloudStack.   Adding a VMware datacenter to CloudStack creates this custom attribute and sets it to "true".  CloudStack will check for this attribute and value and show an error on the management log if it attempts to connect to a vCenter server which does not have this attribute set.  While the property can be edited using the full vSphere client, it must be created using PowerCLI utility.

Using the PowerCLI utility do the following:

Connect-VIServer <VCENTER_SERVER_IP>

(you'll be prompted for a username and password with sufficient permissions for the vCenter Server)

New-CustomAttribute -Name "cloud.zone" -TargetType Datacenter    Get-Datacenter <DATACENTER_NAME> | Set-Annotation -CustomAttribute "cloud.zone" -Value true

Reconnecting the ESXi Hosts to CloudStack

CloudStack will now be connected to the vCenter server, but the hosts will still probably all be marked as down.  This is because vCenter uses an internal identifier for hosts, so that names and IP addresses can change, but it still can keep track of which host is which.

This identifier appears in two places in CloudStack's database; The 'guid' in the cloud.host table and also in the cloud.host_details table against the 'guid' value. The host table can be queried as follows:

SELECT id,name,status,guid FROM cloud.host WHERE hypervisor_type = 'VMware' AND status != 'Removed';

The guid takes the form:

HostSystem<HOST_ID>@<VCENTER_NAME>

You can find the new Host ID either though the PowerCLI again, or via a utility like Robware's RVTools.  RVTools is a very light-weight way of viewing information from a vCenter. It also allows export to csv so is very handy for manual reporting and analysis. The screen grap below shows a view called vHosts; note the Object ID.  This is vCenter's internal ID of the host.

Through PowerCLI you would use:

Connect-VIServer <VCENTER_SERVER_IP>;    Get-VMHost | ft -Property Name,Id -autosize

Now you can update the CloudStack tables with the new host ids for each of your hosts.

cloud.host.guid for  host '10.0.1.32' on vCenter '10.0.1.11' might previously have read:

HostSystem:host-99@10.0.1.11

should be updated to:

HostSystem:host-10@10.0.1.11

Remember to check the host and host_details table.

Once that you have updated the entries for all of the active hosts in the zone, you can start the management service again now and all of the hosts should reconnect, as their pointers have been fixed in the CloudStack database.

Summary

This article explains how to re-insert a lost or failed vCenter server into CloudStack.  This blog describes the use of PowerCLI to both set and retrieve values from vCenter.

This process could be used to migrate an appliance based vCenter to a Windows based environment in order to support additional capacity as well as replacing a broken vCenter installation.

About The Author

Paul Angus is VP Technology & Cloud Architect at ShapeBlue, The Cloud Specialists. He has designed and implemented numerous CloudStack environments for customers across 4 continents, based on Apache Cloudstack, Citrix Cloudplatform and Citrix CloudPortal.

When not building Clouds, Paul likes to create Ansible playbooks that build clouds

----

Shared via my feedly reader

Sent from my iPhone