A tale about open source testing: part 1 – autotest

For quite a while, I’ve been wanting to write about the current state of open source testing. I want to point out some recent developments on this particular area, and what we can do to build a roadmap towards a bigger goal.

But what would be that goal?

Imagine a scenario when one GNU/Linux distribution team needs a system to perform basic regression testing on their “product”, which is the distro itself. The process of doing this regression testing on several components of a distro currently involves a lot of manual work. Most distributions can’t afford having large testing groups, and even when they can have dedicated test engineers, their time can be better used if they focus on analyzing eventual problems and failures, instead of mechanically preparing machines and running the tests there.

Now imagine a fully automated and open sourced system that allows any organization to setup a test farm in such a way that, after a few minutes the isos are ready, several batteries of tests are already running on an automated way in several machines, producing results that will be stored on a database, allowing test engineers to analyze failures, open bugs and help the development team to fix the bugs, instead of spending way too much time on the test execution itself.

Sounds good, isn’t it? And it’s closer to its completion than we think…

Project autotest: building the road to a better open source testing

Those that already had the opportunity to work on open source software test teams know that the deadlines are tight, the test buckets are large and the project managers are allways paying close attention to what’s going on. So every process improvement that leads to more time to actually find and investigate bugs is a very good investiment.

On projects that have a rapid development pace, the process of validation and testing (both performance and regression) becomes more and more difficult and critical. One canonical example of a hugely successful open source project is the linux kernel. Daily lots of code are added to the main linux tree, and keeping up with this frantic development pace is a hard task to anybody.

Probably that’s why the autotest project was started out by kernel hackers: they needed tools to do fully automated testing of the linux kernel, so problems can be spotted early and fixed. Building on the experience of systems, the autotest project aims to be a flexible test framework, that comprises a test harness a, test API and results storage and analysis.

A very good point made by the autotest developers and maintainers is that software testing isn’t about running tests. It’s about finding and fixing problems with software. Running tests is the trivial part of the process, and it’s better to hand this responsibility to computers, that can do these tasks faster and more reliably.  So the autotest design goals (see more on the autotest design goals page) reflect this need:

  • Maintainability: All testing infrastructure and API is consistent easy to understand, so it’s easy to spot bugs on the system itself.
  • Modularity: Core functionality is not tied to the test modules, so adding tests won’t break the whole system. Also, corporations can write modules for proprietaty tests if needed.
  • Good error handling: On an automated test execution environment, well thought and meaningful results / error handling are very important. The autotest API tries to make it easy for test developers to do error handling.

More information about the design goals can be found on the autotest wiki page.

Autotest components overview

Autotest comprises several modules loosely coupled. We can combine them to create some serious test instrumentation:

  • client: This module is installed on the test machine, and takes care of setting up the environment for the tests and run them (in a nutshell). You can do fairly sophisticated stuff on the client side, including parallel execution of tests, profiling and rebooting. There’s a rich API that developers can use to automate their own tests. One of the key points of autotest is that the test description language is also python. You don’t need to learn yet another restrictive language to do test control. You could get started with autotest by just installing the client on a machine and start running simple control files
  • server: coordinates the execution of tests on multiple machines. You can do scarily powerful things on control file servers, that are also (unsurprisingly) python programs. We even have fancy stuff like automatic ssh key setup on the test hosts. The server will take care of checking the clients state, installing the client code on them, kickstart the tests and bring the logs back when the tests are finished.
  • scheduler: Controls the execution of several server side instances that are fired up on each ‘test job’.
  • web interface: A simple to use web interface that the test team can use to add machines, and kickstart jobs. Each job kickstarted is sent to the scheduler, that starts the server program instances.
  • tko: It is a backend of result analysis. Once the tests are finished, they’re loaded into a database, and tko is a frontend of that database, that can show test results using an interesting spreadsheet view, like it can currently be seen on the autotest results page.

Of course, this is the ‘nutshell’ description of what we’ve got. We have almost all pieces needed to perform linux distributions testing done very efficiently.

What is lacking then?

Glad you asked 🙂 The missing piece here is a generic (and open source) operating system provisioning system. Provisioning is the term used to describe an operating system installation. Folks that work with Linux distro testing probably found out that installing distros on test machines is very time consuming and repetitive.

The upstream version of autotest does things on the host based on the assumption that it already has an operating system installed. If we can integrate a good open source automated install system to all the components listed, the goal I mentioned on the beginning of this post would be pretty much complete. We’d lower the bar for community distro testing. We could test a lot more in less time, finding bugs early and make the whole distro development process smoother.

So far I found 2 interesting projects that could be used as provisioning systems for autotest, each one of them using different approaches:

  • FAI, the Fully Automatic Installation project. Uses a traditional approach of automating the operating system install program (anaconda, for fedora and red hat, yast for suse, debian installer for debian).
  • System Imager, a system that uses the ‘os image’ concept, that is pretty much a tarball of the root filesystem for the particular distribution. Looks interesting, image based installs are usually faster than other install methods, however, there’s allways the problem: How we are going to generate the install images?

I need to study a lot more about the systems before I can reach any conclusion. If somebody does have thoughts to share, they’ll be welcome.

I’m planning to write other articles about the state of art of open source test when I have time to do it. Let’s see how that goes.


Guessing the environment where your code is executing on

This week I was working on automating some toolchain tests and I’ve realized that we could greatly reduce the amount of setup done by the tester if we do ‘automagical’ guessing of the underlying processor architecture we’re running on, as well as the hardware capabilities (sets of instructions supported). Since then I was researching about good ways to get this information. One particularly interesting I found about were the ELF auxiliary vectors, described on this article.

ELF auxiliary vectors are carriers of information from kernel space to user space. They are loaded into the ELF program execution stack by the kernel ELF loader. The coolest thing is that we can see the contents of these vectors by setting the variable LD_SHOW_AUXV to 1 before executing any program. Let’s say, /bin/true:

lucas@freedomm:~$ LD_SHOW_AUXV=1 /bin/true
AT_SYSINFO: 0xb7f5a400
AT_SYSINFO_EHDR: 0xb7f5a000
AT_HWCAP: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
AT_PHDR: 0x8048034
AT_BASE: 0xb7f5b000
AT_ENTRY: 0x8048a40
AT_UID: 1000
AT_EUID: 1000
AT_GID: 1000
AT_EGID: 1000

So the info provided by this vector is reasonably safe, since it comes from the kernel itself. The main problem: older versions of the 2.6 kernel won’t support all the variables we’re interested in, such as AT_PLATFORM, which reports, unsurprisingly, the processor architecture.

Meanwhile, I’m going with the poor’s man solution: simple parse of /proc/cpuinfo. That’s good enough for now, but if someone wants to show a reasonably robust and prevalent way of getting CPU information and hardware capabilities, please let me know.

King of Mustache 2008

Yesterday we’ve had the true Battle of the Year:  contest where brave moustached men were running for the title of the LTC Brazil King of Mustache 2008. This contest is our celebration of the brave founding fathers of our nation [1], and also a celebration of life and an opportunity to get all the nerds engineers together.

The official website has more information (warning: content in portuguese only). The beautiful photos of the event can be seen on our gallery.

And from now on, let’s all hail our new mustache overlord, Daniel Debonzi. May your mustache be allways hairy, healthy and powerful for the years to come!

I want to break free, I want to breeeak freeee...
The King of Mustache 2008

[1] They alledgedly didn’t have mustaches at all, but I don’t believe on such sick claims, they *must* have had glorious and shiny mustaches :p