For quite a while, I’ve been wanting to write about the current state of open source testing. I want to point out some recent developments on this particular area, and what we can do to build a roadmap towards a bigger goal.
But what would be that goal?
Imagine a scenario when one GNU/Linux distribution team needs a system to perform basic regression testing on their “product”, which is the distro itself. The process of doing this regression testing on several components of a distro currently involves a lot of manual work. Most distributions can’t afford having large testing groups, and even when they can have dedicated test engineers, their time can be better used if they focus on analyzing eventual problems and failures, instead of mechanically preparing machines and running the tests there.
Now imagine a fully automated and open sourced system that allows any organization to setup a test farm in such a way that, after a few minutes the isos are ready, several batteries of tests are already running on an automated way in several machines, producing results that will be stored on a database, allowing test engineers to analyze failures, open bugs and help the development team to fix the bugs, instead of spending way too much time on the test execution itself.
Sounds good, isn’t it? And it’s closer to its completion than we think…
Project autotest: building the road to a better open source testing
Those that already had the opportunity to work on open source software test teams know that the deadlines are tight, the test buckets are large and the project managers are allways paying close attention to what’s going on. So every process improvement that leads to more time to actually find and investigate bugs is a very good investiment.
On projects that have a rapid development pace, the process of validation and testing (both performance and regression) becomes more and more difficult and critical. One canonical example of a hugely successful open source project is the linux kernel. Daily lots of code are added to the main linux tree, and keeping up with this frantic development pace is a hard task to anybody.
Probably that’s why the autotest project was started out by kernel hackers: they needed tools to do fully automated testing of the linux kernel, so problems can be spotted early and fixed. Building on the experience of systems, the autotest project aims to be a flexible test framework, that comprises a test harness a, test API and results storage and analysis.
A very good point made by the autotest developers and maintainers is that software testing isn’t about running tests. It’s about finding and fixing problems with software. Running tests is the trivial part of the process, and it’s better to hand this responsibility to computers, that can do these tasks faster and more reliably. So the autotest design goals (see more on the autotest design goals page) reflect this need:
- Maintainability: All testing infrastructure and API is consistent easy to understand, so it’s easy to spot bugs on the system itself.
- Modularity: Core functionality is not tied to the test modules, so adding tests won’t break the whole system. Also, corporations can write modules for proprietaty tests if needed.
- Good error handling: On an automated test execution environment, well thought and meaningful results / error handling are very important. The autotest API tries to make it easy for test developers to do error handling.
More information about the design goals can be found on the autotest wiki page.
Autotest components overview
Autotest comprises several modules loosely coupled. We can combine them to create some serious test instrumentation:
- client: This module is installed on the test machine, and takes care of setting up the environment for the tests and run them (in a nutshell). You can do fairly sophisticated stuff on the client side, including parallel execution of tests, profiling and rebooting. There’s a rich API that developers can use to automate their own tests. One of the key points of autotest is that the test description language is also python. You don’t need to learn yet another restrictive language to do test control. You could get started with autotest by just installing the client on a machine and start running simple control files
- server: coordinates the execution of tests on multiple machines. You can do scarily powerful things on control file servers, that are also (unsurprisingly) python programs. We even have fancy stuff like automatic ssh key setup on the test hosts. The server will take care of checking the clients state, installing the client code on them, kickstart the tests and bring the logs back when the tests are finished.
- scheduler: Controls the execution of several server side instances that are fired up on each ‘test job’.
- web interface: A simple to use web interface that the test team can use to add machines, and kickstart jobs. Each job kickstarted is sent to the scheduler, that starts the server program instances.
- tko: It is a backend of result analysis. Once the tests are finished, they’re loaded into a database, and tko is a frontend of that database, that can show test results using an interesting spreadsheet view, like it can currently be seen on the autotest results page.
Of course, this is the ‘nutshell’ description of what we’ve got. We have almost all pieces needed to perform linux distributions testing done very efficiently.
What is lacking then?
Glad you asked :) The missing piece here is a generic (and open source) operating system provisioning system. Provisioning is the term used to describe an operating system installation. Folks that work with Linux distro testing probably found out that installing distros on test machines is very time consuming and repetitive.
The upstream version of autotest does things on the host based on the assumption that it already has an operating system installed. If we can integrate a good open source automated install system to all the components listed, the goal I mentioned on the beginning of this post would be pretty much complete. We’d lower the bar for community distro testing. We could test a lot more in less time, finding bugs early and make the whole distro development process smoother.
So far I found 2 interesting projects that could be used as provisioning systems for autotest, each one of them using different approaches:
- FAI, the Fully Automatic Installation project. Uses a traditional approach of automating the operating system install program (anaconda, for fedora and red hat, yast for suse, debian installer for debian).
- System Imager, a system that uses the ‘os image’ concept, that is pretty much a tarball of the root filesystem for the particular distribution. Looks interesting, image based installs are usually faster than other install methods, however, there’s allways the problem: How we are going to generate the install images?
I need to study a lot more about the systems before I can reach any conclusion. If somebody does have thoughts to share, they’ll be welcome.
I’m planning to write other articles about the state of art of open source test when I have time to do it. Let’s see how that goes.