Re: testing alpha toolchain modules
# from David Golden # on Tuesday 09 September 2008 16:00: >This would be better if I finally updated CPAN::Reporter::Smoker to >take an arbitrary list of distributions instead of just working >backwards through CPAN and if you had a list of distributions with >Build.PL. Indeed I do have such a list, and a cpan mirror from which to generate them. http://www.nntp.perl.org/group/perl.qa/2008/09/msg11418.html I suppose I could make that a cron job or somehow link it to the updates. Incidentally, does anyone know where Randy Sims and his META stats stuff has gone? --Eric -- Consumers want choice, consumers want openness. --Rob Glaser --- http://scratchcomputing.com ---
Re: testing alpha toolchain modules
Hi Eric > Yes, Module::Build needs better tests. It also needs somebody with the > time to write them. (If Devel::Cover worked, I imagine it would tell > me that the coverage is rather low.) Is someone available for a (sort of) Perl micro-grant to write such tests? I could help fund it. -- Ron Savage [EMAIL PROTECTED] http://savage.net.au/index.html
Re: testing alpha toolchain modules
On this thread, Eric asked how to run a "scientific" style test. Here's an idea based on CPAN Testers tools -- though not necessarily CPAN Testers output. a) Configure CPAN::Reporter to use the Test::Reporter::Transport::File transport -- this will save reports into a directory b) Run CPAN::Reporter::Smoker -- this will smoke test all of CPAN in reverse order of upload c) Install alpha M::B d) Change directory for output e) Run CPAN::Reporter::Smoker again f) Compare output in the two directories. This would be better if I finally updated CPAN::Reporter::Smoker to take an arbitrary list of distributions instead of just working backwards through CPAN and if you had a list of distributions with Build.PL. -- David
Re: testing alpha toolchain modules
> On Tue, 9 Sep 2008 02:27:09 -0700, Eric Wilhelm <[EMAIL PROTECTED]> said: >> I have said it, I have (well, CPAN::Testers::ParseReport has) >> downloaded 56000 reports from >> http://www.nntp.perl.org/group/perl.cpan.testers/ > No, I meant what *testers*. I could tell you but you could find out yourself if you would invest the same amount of time in analysing the results. Sorry, I'm running out of time with this MB release. Schwern is already asking for result for the next Test-Simple alpha. > How are the alpha versions getting > installed? Is it manually, via some option in the automated smoke > tools, or what? I have been under the impression that alpha > dependencies never got installed automatically. Every tester has his own strategy. Mine is to install dev releases and get them tested as such. >> If dev releases pummel other authors it's a call for better tests. If >> your tests are good, then release early, release often and watch the >> results on cpantesters. The point of cpantesters for toolchain >> modules: they may not only watch their own but all test results where >> they might be involved. > How does this process work? If I release an alpha of M::B with a bug, > how long before that irritates, distracts, and confuses a bunch of > other authors? Meanwhile, I have to watch test results for 2000+ other > dists to find it? You can answer this one yourself, right? > What is triggering the testing of dists that use M::B though? Is it > only a newly uploaded dist? How can I tell you. Cpantesters are volunteers and may start today and quit tomorrow. -- andreas
Re: testing alpha toolchain modules
# from Andreas J. Koenig # on Tuesday 09 September 2008 00:09: >OK, I walk you through them. First off, there are ten cases in the >file I sent you. >... > So the above is a case where it's impossible to judge without > looking at the report but at the same time we cannot have any > expectations about a single event when the previous outcome was > diverse. Let's call it a case DUNNO. > >CGI-Application-Plugin-ValidateRM-2.2 0.2808 FAIL 18 >CGI-Application-Plugin-ValidateRM-2.2 0.2808_03 FAIL 2 > > Seems like the exact right behaviour. Let's call it a case OK. > ... >So to sum up, we have found that two of the ten support the view that >_03 is doing fine, one appears to be against but is proved wrong, so >seven remaining are simply DUNNOs that we can ignore because they do >not indicate that we have to doubt. > > > Did you chase-down several of those? > >No. The judgement makes sense assuming an even scatter of machine profiles and etc. If it is not actually possible to corelate 0.2808_03 results to 0.2808 results, then I suppose judgement is all we can get. (Being the sort that I am, I would feel a lot better with some form of 1:1 comparisons.) >If somebody with strong statistics fu can measure the trustworthyness >of the data in fovor of a releasse, please speak up. As something to consider for cpantesters 2.0, the ability to analyze this sort of thing without statistics would be very useful. > > 2. Where are these reports coming from? > >I have said it, I have (well, CPAN::Testers::ParseReport has) >downloaded 56000 reports from >http://www.nntp.perl.org/group/perl.cpan.testers/ No, I meant what *testers*. How are the alpha versions getting installed? Is it manually, via some option in the automated smoke tools, or what? I have been under the impression that alpha dependencies never got installed automatically. >If dev releases pummel other authors it's a call for better tests. If >your tests are good, then release early, release often and watch the >results on cpantesters. The point of cpantesters for toolchain >modules: they may not only watch their own but all test results where >they might be involved. How does this process work? If I release an alpha of M::B with a bug, how long before that irritates, distracts, and confuses a bunch of other authors? Meanwhile, I have to watch test results for 2000+ other dists to find it? What is triggering the testing of dists that use M::B though? Is it only a newly uploaded dist? Yes, Module::Build needs better tests. It also needs somebody with the time to write them. (If Devel::Cover worked, I imagine it would tell me that the coverage is rather low.) If I had the time, before a release I would run the M::B tests on multiple platforms and perl versions, then for each of those run through a group of installs for dists known to use Module::Build with known results from a previous run. Those results could be pass or fail -- the metric is whether the same dist does exactly the same thing (e.g. build ok and fail test X, etc) with both versions of M::B. That would be what I would consider a controlled test with quantifiable results. Granted, you cannot prove a negative, but the scientific method would give me a lot more confidence than "probably is okay". Thanks, Eric -- I arise in the morning torn between a desire to improve the world and a desire to enjoy the world. This makes it hard to plan the day. --E.B. White --- http://scratchcomputing.com ---
