LinuxSA Mailing list archives

  From: Christopher Yeoh <cyeoh@samba.org>
  To  : Richard Sharpe <rsharpe@richardsharpe.com>
  Date: Tue, 2 Dec 2003 11:38:03 +1100

Re: Linux.Conf.Au 2004 FIXIT's - You can run one!

At 2003/11/30 14:33-0800  Richard Sharpe writes:
> Many of these tests are automatable, and the goal would be to automate 
> them all.
> 
> However, running these kinds of tests requires several things:
> 
> 1. Hardware. Sometimes lots of hardware. Individuals might not be able
>    afford the required hardware, but organizations like OSDL might.
> 
> 2. Tests. In order to test, you need tests, pure and simple.
> 
> 3. A testing framework that allows the tests to be run in an automated 
>    fashion, and which allows logs of passed, failed, not run, etc tests
>    to be kept, perhaps in a database, and that allows new tests to be
>    easily developed and contributed. It should also allow large tests
>    to be developed. That is, tests that require more than one client
>    and perhaps even, more than one machine under test.
> 
> So, the thing that I think needs fixing in the Open Source Software 
> movement is testing.

Having spent the last few years mostly working on testing I too have
been thinking about this for a while. A lot of projects are building
their own autobuilders and regression testers and I think there is
room for a generic framework to help support this.

Autobuilders are important - the project I work on formally supports 6
architectures with more on the way and just getting people access to
all of the various machines can be difficult. We now do automated
nightly builds and this has been really good at picking up arch
specific problems quickly. We also have started building regression
tests in (regression tests for tests!)

I also use a variant of the autobuilder to make releases packages -
manually building on lots of different machines everytime we wanted to
do a release was starting to take up a significant amount of time.

Automation of tests is very important. If they're not run
automatically they tend to not get run at all - eg I've had a quick
look at the state of POSIX compliance for 2.6 and its not particularly
pretty.

> DejaGNU seems to have a good start for a testing framework. See:
> http://www.delorie.com/gnu/docs/dejagnu/dejagnu.html#SEC_Top

Other options are TET (used for UNIX and LSB certification testing -
http://tetworks.opengroup.org) and QMTest
http://www.codesourcery.com/qm/qmtest which the libstdc++ people have
been looking at moving to (currently they use DejaGNU).

One thing that you do want out of a testing framework is for it to be
POSIX conforming (1003.3). One aspect of this is that it does more
than just return pass/fail, but has finer grained classifications such
as being unable to initiate the test, or the test failed in an
unexpected manner. 

Another important property is that it produces the results in machine
parseable format. This makes it *much* easier to detect regressions
and compare results between different test runs over time or different
machines.

> I think it will make life much easier for those wanting to adopt Open 
> Source software if they can point to a [large] battery of tests that the 
> software has survived, and creating such a framework and battery of tests 
> might well be, at this stage, the biggest single contribution people can 
> make to Open Source software.
> 

Agreed. Its just a part of open source software development that is
really hard to attract people to work on.

Chris
-- 
cyeoh@au.ibm.com
IBM OzLabs Linux Development Group
Canberra, Australia

-- 
LinuxSA WWW: http://www.linuxsa.org.au/ IRC: #linuxsa on irc.freenode.net
To unsubscribe from the LinuxSA list:
  mail linuxsa-request@linuxsa.org.au with "unsubscribe" as the subject