LinuxSA Mailing list archives

  From: Daryl Tester <Daryl.Tester@iocane.com.au>
  To  : Richard Sharpe <rsharpe@richardsharpe.com>
  Date: Mon, 01 Dec 2003 21:18:05 +1030

Re: Linux.Conf.Au 2004 FIXIT's - You can run one!

Richard Sharpe wrote:

> Here is an idea for a very ambitious project that I think the Open Source
> Software movement needs to tackle ...

While I'm not heavily into XP (Extreme Programming, not the 2 bit OS that
post-dates it), I'm starting to dabble with Test Driven Development, which
is a methodology developed by one of XP's designers, Kent Beck.  I've always
been a fan of regression testing anyway, and a big part of TDD requires you
to create an automated test case up-front before you write the code, which
forces some interesting characteristics into your software:

1)  You wind up with a large number of automated test cases, which allows
    you to make changes and test against the code without reduced fear
    that you've broken something (well, you get that with regression
    testing anyway, but you don't put off the test until after the event).
2)  Because the interfaces are designed to be driven by these external
    driver programs, these interfaces have a tendency to be a lot cleaner
    than if they were just used internally.  As the TDD link below states,
    you've just written the code for two clients, not one.  Adding a third
    client will be twice as easy.

Python has a large number of regression tests that you can run against it
when compiled from source to check that you have a "clean compile", and this
is used for automated testing on the "Snake Farm", which is a cluster of
oddball Unix variants specifically set up for testing Python.  It could
be easily extended to actually build the binaries from there, although
I don't believe they currently do so.

For those that care, <http://www.c2.com/cgi/wiki?TestDrivenDevelopment>;
is a good starting point (caution - I find this wiki very addictive).

>   - The software must build on a number of platforms and in a number
>     of ways (release, debug, lint, etc) before we can push changes to
>     the trunk.

If you're allowed to divulge, what are you using for revision control?
CVS?

> Many of these tests are automatable, and the goal would be to automate
> them all.

That's my dream.  :-)  I'm having trouble applying it to my current
project, though, which is both real-time and involves some serial port
shenanigans, but I'm certainly trying to partition the software to
make major components of it more testable (Testerable?).  And testing
the device driver that has to be written later on will present its
share of headaches.

> 1. Hardware. Sometimes lots of hardware. Individuals might not be able
>    afford the required hardware, but organizations like OSDL might.

I think I've mentioned it on this list before, but one of my long running
back burner projects has been to use etherboot/PXE systems to be able
to build a box into "any" configuration, where any may be e.g. some
version of RedHat with an arbitrary set of patches applied, purely to
be able to compile/test software for that platform.  The open OS' have
been fairly straightforward, it's the Win32 systems that's been giving
me the proverbials :-/.

> 3. A testing framework that allows the tests to be run in an automated
>    fashion, and which allows logs of passed, failed, not run, etc tests
>    to be kept, perhaps in a database, and that allows new tests to be
>    easily developed and contributed. It should also allow large tests
>    to be developed. That is, tests that require more than one client
>    and perhaps even, more than one machine under test.

Unless the configuration is fairly static, physically configuring the
machines can be a stumbling block (e.g. for a firewall, setting up a
machine that may fluctuate between one or two DMZs still requires human
intervention).

> So, the thing that I think needs fixing in the Open Source Software
> movement is testing.

I think most of the tools are out there already - certainly, there's
more Unit testing frameworks than you can poke a stick at.  I think
a large part of the problem is getting the developers into that
particular mindset.

> I think it will make life much easier for those wanting to adopt Open
> Source software if they can point to a [large] battery of tests that the
> software has survived, and creating such a framework and battery of tests
> might well be, at this stage, the biggest single contribution people can
> make to Open Source software.

Agreed.

On the other hand, I sometimes spend far too much time thinking about (and
doing) this sort of stuff  :-/.


-- 
Regards,
  Daryl Tester,  Software Wrangler and Bit Herder, IOCANE Pty. Ltd.

"Next time, let's screw it up my way first."  -- Jay D. Dyson

-- 
LinuxSA WWW: http://www.linuxsa.org.au/ IRC: #linuxsa on irc.freenode.net
To unsubscribe from the LinuxSA list:
  mail linuxsa-request@linuxsa.org.au with "unsubscribe" as the subject