Validation of the Creedence Alpha [feedly]

----
Validation of the Creedence Alpha
// Latest blog entries

On Monday May 19^th early access to XenServer Creedence builds started from xenserver.org.  The  xenserver.org community has access to XenServer pre-release installation media of an alpha quality and is invited to provide feedback on it.

This blog describes the validation and system testing performed on the first alpha build.

Test Inventory

The XenServer development process incorporates daily automated regression testing complemented by various additional layers of testing, both automated and manual, that are executed less frequently.

In outline, these are the test suites and test cycles executed during XenServer development.

Automated short-cycle regression testing ("BVT") for fast feedback to developers – on every build on every branch.

Automated medium-cycle regression testing ("BST") to maintain quality on team branches

Automated long-cycle system regression test ("Nightly") – on select builds on select branches, aimed at providing wide regression coverage on a daily basis

Automated performance regression test, measures several hundred key performance indicators – run on select builds on select branches on a regular basis

Automated stress test (huge numbers of lifecycle operations on single hosts) – run once per week on average

Automated pool stress test (huge numbers of lifecycle and storage operations on XS pools) - run once per week on average

Automated long-cycle system regression test ("Full regression") – on select builds on select branches, aimed at providing extensive wide test coverage, this comprises a huge number of tests and takes several days to run, frequency of run is therefore one every two weeks on average

Automated large scale stability test (huge numbers of VMs on large XS pools, boot storm and other key 'scale' use-cases – run on demand, usually several times in the run up to a product release and ahead of key internal milestone including deliveries to other Citrix product groups

Automated soak test – run on demand, usually several times in the run up to a product release and ahead of key internal milestone including deliveries to other Citrix product groups, this comprises long-running tests aimed at validating XS over an extended time period

Automated upgrade test – run ahead of key milestones and deliveries to validate upgrade procedures for new releases.

Manual test – exploratory testing using XenCenter, aimed principally at testing edge cases and scenarios that are not well covered by automation, cycles of manual testing are carried out on a regular basis, and ahead of key milestones and deliveries.

Exit Criteria

Each stage of a XenServer release project requires different test suites to have been run "successfully" (usually meaning a particular pass-rate has been achieved and/or failures are understood and deemed acceptable).

However test pass rates are only a barometer of quality – if one test out of a hundred fails then that may not matter, but on the other what if that one test case failure represents a high impact problem affecting a common use-case? For this reason we also use defect counts and impact analyses as part of the exit criteria.

XenServer engineering maintains a high quality bar throughout the release cycle – the "Nightly" automated regression suite comprising several thousand test cases must always achieve over 95%. If it does not, then new feature development stops while bugs are fixed and code reverted until a high pass rate is restored.

The Alpha.next release is a drop from the Creedence project branch that has achieved the following pass rates on the following test suites.:

• ·         Nightly regression  – 96.5%
• ·         Stress –  passed (no pass rate for this suite)
• ·         Pool stress –  passed (no pass rate for this suite)
• ·         General regression –  91.3%

Drops later in the project lifecycle (e.g. Tech Preview) will be subjected to more testing and with more stringent exit criteria.

More Info

For more information on the automation framework used for these tests, please read my blog about XenRT!

----

Shared via my feedly reader

Sent from my iPhone