For the impatient

Download autotest here

https://github.com/autotest/autotest/downloads

Or pick the tarballs automatically generated by github here:

https://github.com/autotest/autotest/tarball/0.14.0

Changes

API cleanup

Autotest 0.14.0 brings major new API changes over the previous 0.13.X series.
For a long time, we wanted to clean up the API namespace of unneeded _lib
prefixes to the autotest libraries. The main changes made were:

* No more autotest/client/bin dir, so no more autotest_lib.client.bin namespace.
Now all the main libs and entry points are at the top level client.
* API: autotest_lib -> autotest
* API: autotest.client.common_lib -> autotest.client.shared

As an example, here’s how the imports on client/bin/job.py used to be on pre-0.14:

from autotest_lib.client.bin import client_logging_config
from autotest_lib.client.bin import utils, parallel, kernel, xen
from autotest_lib.client.bin import profilers, boottool, harness
from autotest_lib.client.bin import config, sysinfo, test, local_host
from autotest_lib.client.bin import partition as partition_lib
from autotest_lib.client.common_lib import base_job
from autotest_lib.client.common_lib import error, barrier, log, logging_manager
from autotest_lib.client.common_lib import base_packages, packages
from autotest_lib.client.common_lib import global_config
from autotest_lib.client.tools import html_report

And that’s how it looks on 0.14.0:

from autotest.client import client_logging_config
from autotest.client import utils, parallel, kernel, xen
from autotest.client import profilers, boottool, harness
from autotest.client import config, sysinfo, test, local_host
from autotest.client import partition as partition_lib
from autotest.client.shared import base_job
from autotest.client.shared import error, barrier, log, logging_manager
from autotest.client.shared import base_packages, packages
from autotest.client.shared import global_config
from autotest.client.tools import html_report

The conversion should be mostly automatic for most users.

Entry names change

In order to ease discoverability on a system wide install, some widely used
entry point names changed names. By entry point, we mean the autotest high
level utility programs. Check out the name changes:

1. client/bin/autotest -> client/autotest-local
2. server/autoserv -> server/autotest-remote
3. cli/atest -> cli/autotest-rpc-client
4. scheduler/monitor_db.py -> scheduler/autotest-scheduler
5. scheduler/monitor_db_babysitter.py -> scheduler/autotest-scheduler-watcher

Boottool

Autotest used a perl based tool to manage bootloader entries, but the
tool did not support grub2. So when Fedora started to use grub2, we took
the opportunity to rewrite the whole thing and ditch the perl based boot
tool. As a side effect, now it supports installing kernels on newer
Ubuntu too (and well, Opensuse and what else). It is based on the open source
project grubby, the tool used by anaconda and therefore, Fedora and RHEL distros
to manipulate boot entries.

Git history rewrite

The git repo history was rewritten to properly follow git authorship
conventions. This effort reveals over 200 unique authors that did contribute
to autotest during the last 6 years. May we have many years more with an active
and healthy development community!

What’s next?

With the series of cleanups and improvements, here are the next steps:

- Port of the rpc server application to Django 1.4
- Documentation improvements and API documentation generation.
- Further cobbler-autotest integration
- Separation of tests and core autotest code
- Make the autotest test runner to be smarter, more tight in terms of output

You can also help, please join the mailing lists and happy hacking!

It took a while to finally take the decision to move autotest entirely to git.  When we did, we briefly discussed doing a proper history rewrite, to eliminate old svn cruft (the hideous git-svn-id: lines on the commit messages) and assign authorship properly (on git-svn, the committer is considered the patch author). It’s fair enough that git-svn behaves this way, since there’s no reliable way to tell what is the original author of the patch, since each project does the copyright assignment on a different way.

On autotest, in the Old Days (TM), this was done by means of a From: field on the commit message, and late by Signed-off-by. So the mirror did not contain reliable version information to tell the authors, by, say, using git shortlog -s. This is the original mess the repo was:

$ git shortlog -s
     1  00:18.584417Z
     1  Alex Jia
    22  Amos Kong
     1  Aneesh Kumar K.V
    44  Chris Evich
    81  Cleber Rosa
     1  Daniel Veillard
     3  David Greenberg
     1  Eduardo Habkost
     2  Guannan Ren
     1  Jerry Tang
     1  Jiri Zupka
    19  Jiří Župka
     4  Lei Yang
     1  Liu Sheng
     4  Lubos Kocman
   331  Lucas Meneghel Rodrigues
    28  Lukas Doktor
     6  Madhuri Appana
     3  Martin Krizek
     6  Miroslav Rezanina
    13  Nishanth Aravamudan
     1  Onkar N Mahajan
     2  Pavel Hrdina
     1  Philipp Seiler
    27  Qingtang Zhou
     3  Quentin Deldycke
     1  Satheesh Rajendran
     1  Steve
     5  Steve Conklin
     3  Thomas Jarosch
    16  Vinson Lee
     1  Wei Yang
     1  Wenyi Gao
     9  Yiqiao Pu
     1  Yu Mingfei
   154  apw
     9  ericli
     8  guyanhua
   454  jadmanski
   138  jamesren
  1380  lmr
  2450  mbligh
     1  pradeep
     2  root
   795  showard
    12  tangchen

Now, how to solve this? Considering the complete lack of a rigid standard used across the commits, it was clear I’d have to use several passes of git filter-branch vodoo.

So, in the first pass, I wanted to look for signed-off-by: entries, extract the names and authors, and rewrite the commits. If no signed-off-by: name could be successfuly obtained, fall back to the old author. If the old author happens to be one of the (few) commiters of the old svn repo, use a conversion table to replace the names:

mbligh=Martin Bligh <mbligh@google.com>
jadmanski=John Admanski <jadmanski@google.com>
ericli=Eric Li <ericli@google.com>
apw=Andy Whitcroft <apw@shadowen.org>
jamesren=James Ren <jamesren@google.com>
lmr=Lucas Meneghel Rodrigues <lookkas@gmail.com>
showard=Steve Howard <showard@google.com>

Then the following script was created:

 #!/bin/bash

 git filter-branch -f --env-filter '

 get_name_authors_txt () {
     fail=0
     grep $1 /tmp/authors.txt >> /dev/null || fail=1
     if [ $fail -eq 0 ];
     then
       lname=$(grep "^$1=" "/tmp/authors.txt" | sed "s/^.*=\(.*\) $/\1/")
       if [ -n "$lname" ]
       then
           echo "$lname"
       else
           echo "$GIT_COMMITTER_NAME"
       fi
     else
       echo "$GIT_COMMITTER_NAME"
     fi
 }

 get_email_authors_txt () {
     fail=0
     grep $1 /tmp/authors.txt >> /dev/null || fail=1
     if [ $fail -eq 0 ];
     then
       lemail=$(grep "^$1=" "/tmp/authors.txt" | sed "s/^.*=.* $/\1/")
       if [ -n "$lemail" ]
       then
           echo "$lemail"
       else
           echo "$GIT_COMMITTER_NAME"
       fi
     else
       echo "$GIT_COMMITTER_EMAIL"
     fi
 }

 get_name () {
     name=$(git log $GIT_COMMIT -1 | grep "Signed-off-by:" | head -1 | sed "s/^.*:\(.*\) $/\1/" | grep -v "Signed-off-by:")
     if [ -n "$name" ]
     then
         echo "$name"
     else
         get_name_authors_txt $1
     fi
 }

 get_email () {
     email=$(git log $GIT_COMMIT -1 | grep "Signed-off-by:" | head -1 | sed "s/^.*:.* $/\1/" | grep -v "Signed-off-by:")
     if [ -n "$email" ]
     then
         echo "$email"
     else
         get_email_authors_txt $1
     fi
 }

 GIT_AUTHOR_NAME=$(get_name $GIT_COMMITTER_NAME) &&
     GIT_AUTHOR_EMAIL=$(get_email $GIT_COMMITTER_NAME) &&
     GIT_COMMITTER_NAME=$GIT_AUTHOR_NAME &&
     GIT_COMMITTER_EMAIL=$GIT_AUTHOR_EMAIL &&
     export GIT_AUTHOR_NAME GIT_AUTHOR_EMAIL &&
     export GIT_COMMITTER_NAME GIT_COMMITTER_EMAIL
 '

git filter-branch f --msg-filter 'sed -e "/^git-svn-id:/d"'

So, the basic idea is to try find the first Signed-off-by: line of the patch, extract the author, provided it is in the standard format, and set that to the author name, same process for the email.

This, for the first pass, greatly cleaned up the state of the commits, by assigning proper names and then getting rid of the git-svn-id: tags.

As a matter of fact, I’ve used this same script with small variations over and over to get to the (mostly) complete cleanup, and now things look a lot better:

$ git shortlog -s | wc -l
202

And obviously, this blog post is a way to remember, and hopefully it might be useful to others. About rewriting the history, I know, that’s bad, but obviously I’ll avoid doing it again, and now people can extract stats from the code base more easily now.

[1] And in my humble opinion they should love it, it’s a pretty great program

Gamification

Gamification is a very interesting idea for a community based project. The new tagger page is making use of these concepts, so the more tags people tag, the more badges they earn. We even might have a “Fedora Contributor Network”, where we’ll have ranking boards for Fedora based on their accomplishments. Well done, guys!

Software centers, App Markets or whatever

The concept of a place that is a one stop shop for your software needs is not a new idea, and definitely organizations have been building successful models around it. I for one would like to see such a software center in Fedora.

Spot argues that it is a successful model for people looking for earning money out of it, but I beg to differ: In Fedora, we are interested in a subset of these goals: we want people to browse the available apps in our repos [1], and install that software with the least hassle possible (ideally with 1 click), with great search capabilities, and possibly reviews and other cool things around it.

Anyway, I understand why the team didn’t come after this model in the first pass. I’d like to see things going on this direction, although such an undertaking is huge work, and brings obvious concerns over scalability and maintainability. Still, good to think about it…

[1] We don’t even have to introduce the concept of ‘repos’ to people, making things easy to grasp to less-technically inclined people.

• Autotest uses a “self contained directory” design
• Makes sense for its purposes, but no go for packagers
• Web interface depends on gwt, which is hard to package properly

As a result, it’s been a long ongoing task with upstream to make it more packaging friendly. But, we’re almost there. I promise

I’ve started to assemble my TODO list for the FUDCon days. It shouldn’t be that difficult to get this done between one presentation and another:

• https://github.com/autotest/autotest/issues/88
• https://github.com/autotest/autotest/issues/89
• https://github.com/autotest/autotest/issues/140

I hope to come up with more stuff, let’s see…

• KVM feature, feature process + deadlines (I’m interested in this one)
• F17 schedule
• Build boxes
• Test team status – bugzilla review with the IBM team
• Road map and release engineering

It’s gonna be an interesting conference

I’m getting ready to go to FUDCon Blacksburg next week, where I’ll talk about the current Fedora test automation initiatives, as well as work with the brave folks of the QA team to put all nails needed in the coffin of the task ‘autotest packaging’. I’ll be around, so if you are interested in test automation, let’s talk about it!

See you in about a week,

Lucas

With the recent security incident involving kernel.org, our domains were taken out of DNS as the administrators of the domain were working to fix the security breach, that made our life a bit… hard, to say the least. All the infrastructure we were relying on, our trac instance, that comprised wiki, source code visualization, timeline view, and our patchwork instance, gone.

So, this inconvenient situation triggered the work to fix most of those aforementioned problems. Now we:

• Moved to git
• Moved to a source code hosting facility, so we spent less time maintaining infrastructure and focusing on delivering a kickass code base for our users
• The new infrastructure also has a code review mechanism, better than using patchwork
• An interesting look at how to contribute changes to the project (github pull requests)
• The wiki now is a git repo, with plain files written in reStructuredText
• We added new maintainers for the project and now are in the process

I’m personally very happy with the improvements we’ve implemented, but there’s quite a bit of stuff to be done. So, be welcome to our new project page:

http://autotest.github.com/

We will make autotest.kernel.org to point to this domain, stay tuned