Mark Hindess wrote:
I think it might help this disagreement if we step back and decide
what scenarios for running tests we are trying to optimise.
Disagreement ? What ? On this mailing list ? :-)
Personally, whenever I write tests I'm doing it to optimise the
scenario where a new users comes to the project and does:
1$ svn co ... classlib # or wget/tar if you prefer
2$ cd classlib/make
3$ ant test
I do this because I want the tests to be easy to run by new users
particularly on new platforms. I hope making it this easy means we
get more people running tests and that we get a broader set of results
than we could acheive ourselves.
This scenario becomes slightly less pleasant if, between steps 2 and
3, [*] you have to find your nearest Linux machine and install Apache
httpd, your-favourite-ftp-server, Dante socks server, etc. This
typically means that I'd be inclined to write stub servers to test
against.
But this doesn't mean George is wrong! Because *if* there was a
publically accessible Internet server that already had Apache httpd,
twoftpd (my favourite ftp server this week), Dante socks, etc, then
the scenario I like to optimise becomes possible. The effort of
setting up one hosting server is definitely cheaper than the effort of
implementing stubs - I know because I set up the server George tests
against and it didn't take much time at all.
Agreed : a central, publicly available test server that could be used
for network-related testing would IMHO be advantageous to us all. I
wonder if such a server can be supplied through the auspices of the ASF
? Incidentally, it took me only a small amount of time to set up my
laptop for running the tests on locally.
Having one server means we wont get 1000's of users asking Apache
httpd, twoftpd, Dante socks configuration questions on our mailing
lists. It also means we have a way to see the other half of the
results - that is, the server logs. (Stubs should make this easier
and this cost should be considered too but if we ran the server we
could make this easier.)
George's approach continues to be cheaper even if a few groups have to
set up there own servers - though I think the set of users who "don't
have Internet access but do have a mechanism for getting up to date
Harmony code" should be small and getting smaller. Of course, it
becomes much more expensive if everyone has to do it.
Agreed.
I've never tried using George's approach on past projects because I've
either also been writing server code, that can/should be used by
tests, or because the server was massive and I didn't have the
experience or resources to run a real server. Because of this,
initially, I was inclined to support the implemention of stubs, but
now I'm not so certain.
I think it basically comes down to whether or not we can provide a
central server. I'm not sure it's a simple question but for me this
question is key to this issue.
Or perhaps I'm optimising for the wrong scenario?
Regards,
Mark.
[*] More likely people would run step 3 find it goes horribly wrong
then read the README they ignored initially and discover why. ;-)
