Thursday, 3 June 2010

Qt: open testing infrastructure

I recently wrote a post about how Qt could become a more open project, but after writing it, I was left asking myself what more I could do to help further the process.

As a result, I'd like to introduce you to a project I lovingly title the Qt Community Integration System. Its aim: to automatically build, test, and publish community merge requests.

This is an important step in order to help streamline the process of community contribution and get feedback to individual developers quicker. Qt  internally use a similar system, but theirs is not (yet?) public.

How does it work? There are a number of components:

  • A community Qt repository (why not the 'official' one? read on - this is not a fork)
  • This repository is updated by an integration script, which pulls pristine Qt, applies designated merge requests over the top of it, and then pushes it to the master-proposed branch, if they all merge cleanly.

    These lists of merge requests are currently maintained by

    However, the intention is to give other trusted people, such as those in #qt-labs or who submit frequent merge requests, access to edit these lists and push to this community repository too, so it becomes possible to request integration of a change if you're experienced and trusted.
  • A BuildBot which pulls Qt from the master-proposed branch of this repository, and then proceeds to try to configure, compile, build, and run Qt's extensive test suite, with the help of a custom tool I have purpose-written for this task, unimaginatively called qtestrunner.

    The results of these tests are then published on a website, where they may be viewed by others.
  • This last bit of this puzzle isn't quite finished, but, assuming all of the above steps succeed with no regressions from the previous test run, the master-community branch is updated with the latest contents of the master-proposed branch, as a working snapshot of Qt+community patches.

(Of course, 'master' will eventually be 'all current Qt branches'. I just wanted to pick one to start with.)

So, after I've described all of the above utopia, what's the current status? Well, I've got a lot of the pieces in place, and now I'm fighting problems with Xvfb causing some of the Qt GUI tests to fail *sometimes*, (even though they don't when I run them manually). Once that is solved, I hope to start adding people's access to integration, and start test-integrating merge requests. :)

Hopefully, all this work will help to accelerate the time it takes to send merge requests back for review if they break something, and ease the burden on getting requests merged in (due to requiring a little less testing). Time will tell. But at least I'm doing my bit. Patches welcome. :)

(thanks to John Brooks for copy-editing my appalling English as usual)


  1. Another thing that I would think would help a lot is something like that keeps track of the declarations and calls. It would be more time consuming, but also having the Qt API documentation link each method, signal, slot, etc back to its declaration in the Qt source code would also be very useful. As far as I have been able to find there is a lot of good documentation on how to use Qt, but very little on how it was programmed or any easy way to find where something is in the source doe so you can start modifying it. This creates a much higher barrier to entry than KDE, which at least makes it very easy to find what you are looking for in the source code. If Qt already has stuff like this, it should probably be linked in the gitorious page or something so people can find it.

  2. toddrme2178: you raise a good point. are you on freenode? if so, it might be worth popping into #qt-labs - if you want to lead the way on this, I'm more than happy to help you with it in whatever way I can.