Re: Module::Build 0.2809 release coming, should we test it?
> On Mon, 8 Sep 2008 16:36:00 -0700, Eric Wilhelm <[EMAIL PROTECTED]> said: > # from Andreas J. Koenig > # on Monday 08 September 2008 15:16: >> Since yesterday I have downloaded and analysed ~56000 testreports from >> cpantesters and found ~135 distros that have been tested by both MB >> 0.2808 and 0.2808_03. There is only one result (Test-Group-0.12) that >> looks bad but it turns out to be due to broken Test::More 0.81_01. All >> others suggest that _03 is doing well. > Umm... okay. > 1. I see a lot of m/0.2808_03 +FAIL/ in there. OK, I walk you through them. First off, there are ten cases in the file I sent you. B-Generate-1.130.2808 FAIL 5 B-Generate-1.130.2808 PASS 6 B-Generate-1.130.2808_03 FAIL 1 So the above is a case where it's impossible to judge without looking at the report but at the same time we cannot have any expectations about a single event when the previous outcome was diverse. Let's call it a case DUNNO. CGI-Application-Plugin-ValidateRM-2.2 0.2808 FAIL 18 CGI-Application-Plugin-ValidateRM-2.2 0.2808_03 FAIL 2 Seems like the exact right behaviour. Let's call it a case OK. Devel-LeakTrace-0.05 0.2808 FAIL 43 Devel-LeakTrace-0.05 0.2808 PASS 6 Devel-LeakTrace-0.05 0.2808_03 FAIL 1 It's a DUNNO but likelihood is high that we need not look closer on this one. HTTP-Proxy-0.230.2808 FAIL 8 HTTP-Proxy-0.230.2808 PASS 5 HTTP-Proxy-0.230.2808_03 FAIL 6 HTTP-Proxy-0.230.2808_03 PASS 1 Although it's a DUNNO, the distribution between fail and pass is quite good. Math-BaseCalc-1.0120.2808 FAIL 9 Math-BaseCalc-1.0120.2808 PASS 9 Math-BaseCalc-1.0120.2808_03 FAIL 1 A DUNNO. Metaweb-0.05 0.2808 FAIL 14 Metaweb-0.05 0.2808 PASS 10 Metaweb-0.05 0.2808_03 FAIL 1 DUNNO Parse-BACKPAN-Packages-0.330.2808 FAIL 18 Parse-BACKPAN-Packages-0.330.2808 PASS 8 Parse-BACKPAN-Packages-0.330.2808_03 FAIL 1 DUNNO Template-Plugin-Class-0.13 0.2808 FAIL 6 Template-Plugin-Class-0.13 0.2808 PASS 55 Template-Plugin-Class-0.13 0.2808_03 FAIL 1 DUNNO Test-Group-0.120.2808 PASS 47 Test-Group-0.120.2808_03 FAIL 1 A WHOAA THERE, that seems to indicate that something's wrong. But as I explained in the previous mail, this is due to Test-Simple dev release. Test-JSON-0.06 0.2808 FAIL 15 Test-JSON-0.06 0.2808 PASS 44 Test-JSON-0.06 0.2808_03 FAIL 1 A DUNNO again. So to sum up, we have found that two of the ten support the view that _03 is doing fine, one appears to be against but is proved wrong, so seven remaining are simply DUNNOs that we can ignore because they do not indicate that we have to doubt. > Did you chase-down several of those? No. > Are you saying that having > "some fail" on 0.2808 implies that "some fail" on 0.2808_03 means > no regression, or did you manage to somehow correlate the > 0.2808_03 fails to the same machines sending 0.2808 fails? As explained above, I used judgement. If somebody with strong statistics fu can measure the trustworthyness of the data in fovor of a releasse, please speak up. > 2. Where are these reports coming from? I have said it, I have (well, CPAN::Testers::ParseReport has) downloaded 56000 reports from http://www.nntp.perl.org/group/perl.cpan.testers/ > Again, the subject of false > fails: I would hate for testers to be pummelling other authors with > alpha M::B errors while the M::B maintainers are left blissfully > ignorant. Toolchain maintainers will probably want to use ctgetreports which comes with CPAN::Testers::ParseReport. If dev releases pummel other authors it's a call for better tests. If your tests are good, then release early, release often and watch the results on cpantesters. The point of cpantesters for toolchain modules: they may not only watch their own but all test results where they might be involved. -- andreas
Re: Module::Build 0.2809 release coming, should we test it?
> On Fri, 5 Sep 2008 16:48:37 -0700, Eric Wilhelm <[EMAIL PROTECTED]> said: > http://scratchcomputing.com/tmp/generated_by.module_build.list Since yesterday I have downloaded and analysed ~56000 testreports from cpantesters and found ~135 distros that have been tested by both MB 0.2808 and 0.2808_03. There is only one result (Test-Group-0.12) that looks bad but it turns out to be due to broken Test::More 0.81_01. All others suggest that _03 is doing well. -- andreas count-for-ewilhelm.pl.out.condensed Description: Binary data
Re: Module::Build 0.2809 release coming, should we test it?
# from Andreas J. Koenig # on Monday 08 September 2008 15:16: >Since yesterday I have downloaded and analysed ~56000 testreports from >cpantesters and found ~135 distros that have been tested by both MB >0.2808 and 0.2808_03. There is only one result (Test-Group-0.12) that >looks bad but it turns out to be due to broken Test::More 0.81_01. All >others suggest that _03 is doing well. Umm... okay. 1. I see a lot of m/0.2808_03 +FAIL/ in there. Did you chase-down several of those? Are you saying that having "some fail" on 0.2808 implies that "some fail" on 0.2808_03 means no regression, or did you manage to somehow correlate the 0.2808_03 fails to the same machines sending 0.2808 fails? 2. Where are these reports coming from? Again, the subject of false fails: I would hate for testers to be pummelling other authors with alpha M::B errors while the M::B maintainers are left blissfully ignorant. But those are just observations on the past. I think we're probably ready to ship. Thanks, Eric -- "It works better if you plug it in!" --Sattinger's Law --- http://scratchcomputing.com ---
Re: Module::Build 0.2809 release coming, should we test it?
# from David Cantrell # on Friday 05 September 2008 05:56: >> Perhaps 2286 is still a lot. A one-liner tells me there are 474 >> authors. I wonder if starting with one dist from each author would >> be a useful sampling, since often the weird stuff happens when an >> author found a way to do some undocumented thing with M::B and we >> didn't know about it. Should I split-out two lists that way? > >Please. Here it is with the whole thing and the two parts (once per author and then the rest) http://scratchcomputing.com/tmp/generated_by.module_build.list http://scratchcomputing.com/tmp/generated_by.module_build.onceper http://scratchcomputing.com/tmp/generated_by.module_build.therest Once we get this release out, if anyone is interested in smoking the M::B svn I could easily regen this stuff daily. Thanks, Eric -- Those who cannot remember the past are condemned to repeat it. --George Santayana --- http://scratchcomputing.com ---
Re: Module::Build 0.2809 release coming, should we test it?
> > Just a big long list of AUTHOR/dist-1.23.tar.gz lines would be > > sufficient. > Thanks. Does this work? > http://scratchcomputing.com/tmp/generated_by.module-build.txt Perfect. > Perhaps 2286 is still a lot. A one-liner tells me there are 474 > authors. I wonder if starting with one dist from each author would be > a useful sampling, since often the weird stuff happens when an author > found a way to do some undocumented thing with M::B and we didn't know > about it. Should I split-out two lists that way? Please. > Now, of course setting-up your smokes with MB as the preferred installer > and such is important. > > It would also be necessary to be able to check fails from this against > the previous MB version. Yep. I'll run them all first with the MB that ships with 5.10.0, then upgrade and do it again. Do you have a more recent dev release than what's on the CPAN right now? -- David Cantrell | top google result for "topless karaoke murders" All principles of gravity are negated by fear -- Cartoon Law V
Re: Module::Build 0.2809 release coming, should we test it?
>On Thu, Sep 04, 2008 at 12:04:45AM -0700, Eric Wilhelm wrote: >> My examination of the .meta files in the cpan says [correction: 2286] >> distributions have a META.yml with 'generated_by' citing >> Module::Build. To my knowledge, the testing of pre-release versions >> does not extend to building or testing these [2]k+ distributions, and >> once it ships, the failure reports go to a lot of authors, not M::B >> maintainers. So, that worries me. Does anyone have the ability to >> setup a set of out-of-band tests to avoid spamming everyone else >> with my failures? > >Sure. I'll just set up 5.10.0 and CPAN::Reporter, and disable all > email sending. Then we can just look in its log file to see what > passed and what failed (that will also include any dependencies). > >Would it be sufficient to test, say, a random sample of 300 of those... > >Just a big long list of AUTHOR/dist-1.23.tar.gz lines would be > sufficient. Thanks. Does this work? http://scratchcomputing.com/tmp/generated_by.module-build.txt Perhaps 2286 is still a lot. A one-liner tells me there are 474 authors. I wonder if starting with one dist from each author would be a useful sampling, since often the weird stuff happens when an author found a way to do some undocumented thing with M::B and we didn't know about it. Should I split-out two lists that way? Now, of course setting-up your smokes with MB as the preferred installer and such is important. It would also be necessary to be able to check fails from this against the previous MB version. Please let me know where else I can help. Thanks, Eric -- "It ain't those parts of the Bible that I can't understand that bother me, it's the parts that I do understand." --Mark Twain --- http://scratchcomputing.com ---
Module::Build 0.2809 release coming, should we test it?
Hi all, Module::Build hasn't shipped a proper release for a good while, and a few alphas have gone out since then (including the one in 5.10.0). Now I find myself apparently expected to ship it. My examination of the .meta files in the cpan says 9095 distributions have a META.yml with 'generated_by' citing Module::Build. To my knowledge, the testing of pre-release versions does not extend to building or testing these 9k+ distributions, and once it ships, the failure reports go to a lot of authors, not M::B maintainers. So, that worries me. Does anyone have the ability to setup a set of out-of-band tests to avoid spamming everyone else with my failures? Tools to generate this list of distributions from a cpan mirror could be made, or tell me what would help. Andreas, I think you said you had run a round of tests on the alpha just before this one? With the installer set to MB? Oh, and the latest change is blocking TAP::Harness changing to a "foo ... ok" output format. Statistical snacky bit: the 'generated_by' count on the big three goes: extutils::makemaker: 23338, module::build: 9095, module::install: 8289. Thanks, Eric -- We who cut mere stones must always be envisioning cathedrals. --Quarry worker's creed --- http://scratchcomputing.com ---
