Uncategorized

Integrating autotest with trac testcase management plugin

During the holiday we had here in Brazil I’ve spent some time working on integrating my autotest control files with our trac testcase management plugin. For those that don’t know, we hacked the Trac testcase management plugin [1] to suit our needs, and developed some macros that generate proper test bucket [2] state reports. The plugin can generate the test buckets and neat tables, summary tables and plugins. Every test is mapped to a trac ticket.

Now, if we could integrate our autotest control files to automagically update the test tickets on the event of a pass of failure, that’d be great, isn’t it? Yep. And that’s possible since trac has an xmlrpc server. Python, with its ‘batteries included’ philosophy includes the xmlrpclib module, that allows us to interact with an xmlrpc server. First, you need to install the Trac XMLRPC plugin, and the plugin page is an excellent source on how to install it [3].

After the trac instance is restarted, test if everything is working by doing some tests using the interactive python prompt (I definitely recommend using the ipython python shell). Let’s suppose that the trac instance is running on http://mydomain.foo.com/trac/mytracinstance. The xmlrpc server URL that you’ll have to build looks like:

http://trac_user:trac_password@mydomain.foo.com/trac/mytracinstance

So you could build that URL and connect to the server as follows:

import xmlrpclib
trac_user = "trac_user"
trac_password = "trac_password"
trac_rpc_root = "mydomain.foo.com/trac/mytracinstance/login/xmlrpc"
trac_url = 'http://%s:%s@%s' % (trac_user, trac_password, trac_rpc_root)
trac = xmlrpclib.ServerProxy(trac_url)

At this point we already have a server connection object stablished on trac, so we just need to call remote methods as methods of this connection object. You can get a complete list of methods accessible from this server connection looking at the XMLRPC plugin page. To make sure this connection is working you can call the method ticket.show(), that requires the ticket number. So, assuming that you’ve provided a valid ticket number to it, you’ll get the fields of your ticket on a list that contains a dictionary with the ticket values, like:

In [12]: trac.ticket.get(100)
Out[12]: 
[100,
 1185952767,
 1186675548,
 {'cc': '',
  'component': 'Distro',
  'description': "''''''Test Details:''''''\n\nA highly configurable stress test utility that calls most of the main file system syscalls. Originally developed by SGI for XFS testing.\n\n''''''Expected result:'''''' \n\n0 as return code",
  'keywords': '',
  'machine_name': 'lpar_name',
  'machine_type': 'HV8',
  'milestone': 'DummyDistro',
  'owner': 'lucasmr@br.ibm.com',
  'priority': 'major',
  'reporter': 'lucasmr@br.ibm.com',
  'resolution': 'fixed',
  'status': 'closed',
  'summary': 'TestID: fsstress -- HV8',
  'testcase_bugzilla': '',
  'testcase_result': 'Pass',
  'type': 'testcase'}]

If calling this method works as described, we should be good to go. As one may guess, the method that updates the tickets is ticket.update, as follows:

array ticket.update(int id, string comment, struct attributes={}, boolean notify=False)
  Update a ticket, returning the new ticket in the same form as getTicket().

So you have to pass the ticket id, a string comment (that might for example contain an URL to the test logs), a dictionary with the new ticket fields (for the fields that will be updated). Now, to the autotest control file.

An autotest control file contains fragments of python code, but it can grow to something quite elaborate. The main interface from running tests on a control file is the method job.run_test, a simple autotest control file would be something like:

job.run_test('sleeptest')

The job.run_test() method takes a ‘test name’ as its argument, and returns True if the test passes, False when it fails, so the logic to implement trac reporting is very straightforward:

ticket = 150
message = "[http://myhost.foo.com/results/sleeptest Sleeptest log]"
if job.run_test('sleeptest'):
	trac.ticket.update(ticket, message, \
		{'status': 'closed', 'resolution': 'fixed', \
		'testcase_result': 'Pass'})
else:
	trac.ticket.update(ticket, message, \
		{'status': '', 'resolution': '', \
		'testcase_result': 'Fail'})
 

This way, when the test passes, it will close it and mark it as pass with no intervention, when it fails it only marks it as failed, leaving it open so the test engineer can analyze the failure and came up with a conclusion/bug report.

It was a fun holiday indeed 🙂

[1] We plan on integrating those modifications to the plugin, of course.
[2] By test bucket we mean the sequence of testcases that have to be performed for a single project, let’s say, distro testing.
[3] It’s packaged as a python egg, that could be installed using easy_install or unpacking it and dropping it to your python site_packages directory

Advertisements

3 thoughts on “Integrating autotest with trac testcase management plugin

  1. When will this be included in the test case management plugin? This is exactly what we are looking for to complete the workflow from test cases all the way to automation.

  2. Sumit, unfortunately this got side tracked and I didn’t have time to work on it. I can try to help on whatever is possible to finish this…

  3. Aaah..no problem. I will try to go through the plugin to see if I can figure out a way. We particularly needed to add more fields to the test case to gel with the bug reports we create so a test case failure automatically leads to a comprehensive bug report. Then we wanted to link it with selenium automated tests.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s