This is a companion article for the presentation given during KVM Forum 2012, held in Barcelona. Here’s the link with the slides.
The juggler problem
The biggest problem we have with regards to development of test automation,and achieving reasonable test code coverage on KVM code is similar to the problems found on many projects and organizations, which I believe it could be nicknamed ‘the juggler problem’: Software developers have to handle daily a large number of tasks, ranging from developing new features to reading and debugging existing code.
Then, there’s testing.
It’s hard to write testcases given that you have to implement that brand new algorithm you’ve figured out last week, work on 10 new bugs found by customers, review 3 new large patchsets sent to the upstream mailing list. If the test tools are difficult to use, they’ll end up being dropped on the floor.
Therefore, to avoid the juggler problem, testing tools have to be easy:
- To understand;
- To modify;
- To access new code functionality (do useful stuff with the tests).
If the testing infrastructure does not have the required properties, it is likely that the developer will start rolling out their own testing programs. Since some testing is better than no testing, this is OK.
The problem is when you want to do more complex things.
You’ll start to write functions to do things like migration, hotplug, virtio console, then encapsulate other functionality to it. You’ll start to slowly re-implement the test frameworks. You’ll keep yourself alienated from what your QA team has been developing, and benefit from the code that automates complicated parts of your test workflow.
The solution here is to make the test tools to have the desired properties. In KVM, we do have a codebase being actively developed to cover a lot of testing scenarios, ranging from functional to regression testing, that is being used to run tests against several branches of the KVM tools:
- Upstream code
- RHEL 5 product
- RHEL 6 product
- RHEV product
That codebase was known as KVM Autotest. These days, the name KVM autotest is not entirely appropriate, since it aims to cover more than KVM, but also other virtualization backends, such as libvirt. It can define and handle multiple vms, disks, network cards, run vms, migrate them, hotplug NICs, put the VMs in S3 mode, among others. It’s now known as ‘virt tests’.
Even with all the features, this code base being located inside a separate repo of a larger test framework made general core developer adherence low, and with that came the perception that the test framework is a QA only affair. This certainly is a sub optimal arrangement, and for a long time we’ve been looking for a solution for it.
Resolution features
How to keep the QA functionality of the virt tests? By QA functionality we
mean:
- Comprehensive tests, that involve guest OS install
- Test jobs that involve kernel build, qemu build, install of windows and linux guests
And still provide a fast, unittest like way of executing the tests?
The answer we found was:
- Leverage the autotest modularity, and make the virt tests self contained on a single module, separate from the rest of the framework
- Implement a minimal test harness that allows executing virt tests outside autotest
- Create a test runner that uses this minimal test harness, and outputs only the very minimum amount of information, similar to a unittest execution
- Provide a minimum guest image ready to be used, in order to save bandwidth and time of folks wanting to try the tests.
Modularity
About 18 months ago, when people started asking that KVM autotest could support other virtualization backends, the solution we devised was:
- Create a shared autotest library that any test could use
- Re-factor the kvm test to use this library
- Implement other tests that can use this library. That’s how the libvirt test was born
While the approach is reasonable and worked for the purposes mentioned, it
implied that changes on virt tests mean changes on autotest core, a process that
goes against giving autonomy to developers wanting to implement tests outside
the autotest tree.
So, in order to separate tests from core autotest, the virt tests were all
restructured to be a single test module of autotest. Then it was possible to
separate them from core autotest cleanly.
Minimal test harness
When executed by autotest, the test job is expressed as a control file, a list of operations to execute on an autotest client. Each test runs a lot of code after/before execution, that is autotest housekeeping, log collection, among other things. For people for whom autotest is unnecessary, we had to make the code to execute outside that harness.
So we implemented only the bare minimum methods required to have the code to run outside autotest, and created a small test harness. Just as a POC, we did extract all autotest code needed to run the virt tests outside autotest. Since we had to extract way too many LOC to make things work (about 2k LOCs), we decided to keep a light dependency on autotest, that can be fulfilled by installing an autotest rpm, which is available on Fedora repo (about 2 MB of rpm data).
With this harness, a test runner was implemented, displaying minimal info about what tests are running, their status and running time:
# ./run -t kvm
SETUP: PASS (0.23 s)
DEBUG LOG: /home/lmr/Code/virt-test.git/logs/run-2012-10-15-14.58.24/debug.log
TESTS: 2
kvm.virtio_blk.smp2.virtio_net.JeOS.17.64.boot: PASS (22.87 s)
kvm.virtio_blk.smp2.virtio_net.JeOS.17.64.shutdown: PASS (8.61 s)
With this runner, it is possible to list available tests
# ./run -t kvm --list-tests
Specify which tests to run
# ./run -t kvm --qemu /path/to/my/qemu --tests "boot shutdown"
And provide a specific qemu path to test against
# ./run -t kvm --qemu /path/to/my/qemu
Minimum guest image
In order to have a small guest to run tests on developer’s machines, we did look
into Buildroot (http://buildroot.uclibc.org/) and managed to build a very small
guest to use on virt tests. However, concerns about how to properly manage
the availability of source code used to build this guest, as well as ease of build
and use, made us to reconsider it and take another approach: Getting a minimal
base Fedora system as the minimal guest.
By tweaking a kickstart file used to install Fedora, we managed to get a guest
that is still small, but more complete, easy to recreate, and with no problems
with regards to redistribution. We preferred that approach over buildroot.
Roadmap
Although all those changes greatly improve the usability situation, there are
more areas we want to tackle:
- Being able to run tests written in any language, provided they return 0 upon
- PASS and != 0 upon FAIL.
- Wrap test functionality on scripts/language bindings, so people can use complex functionality automation on their own scripts using different languages
Getting started and contact info
If you are interested in running the virt tests, please clone our virt tests
git repo:
git clone git://github.com/autotest/virt-test.git
Check the testsuite documentation
https://github.com/autotest/virt-test/wiki
And let us know your findings. You can always get in contact with me and Cleber,
the maintainers of the suite:
lmr AT redhat DOT com
cleber AT redhat DOT com
Also, the virt tests have a development mailing list
Virt-test-devel AT redhat DOT com
To subscribe see:
http://www.redhat.com/mailman/listinfo/virt-test-devel
Thanks!
Oh Brave New World! 