On Sat, 23 Feb 2008, James Keenan via RT wrote: > The patch attached, diff.trunk.tcif.txt, addresses both RT 47391 > (t/configure/1*.t tests incorrectly rely on init::defaults_ and RT 47503 > (Remove config::init::defaults From configure tests). It accomplishes > the following:
Thanks for working on this. Alas, it didn't work. I don't have any more time to debug this any further, but I'll share some observations I had: > * In each test, the configuration step is first run just as it will be > by Configure.pl. The results of that step are recorded on disk. The > test file then reads from the disk the results of all previous > configuration steps and uses that in tests for the step currently being > tested. Thus, for the testing of a given configuration step, individual > tests have available to them all the information found in the > Parrot::Configure object at that same point during regular configuration. This sounds to me as if it assumes all the tests will be running in order every time. For example, suppose I have a test failure with test number 200. Does that mean I have to rerun all 199 preceding tests every time I wish to change something in test 200? Conversely, suppose all 200 tests run, but I want to go back and investigate a failure in test 10. The stored results will reflect all 200 tests, so they won't be the same the next time I run test 10. Will that be an issue? More generally: Where is this cache stored? How do we know if it's still valid or not? What invalidates this cache? > * The harness which runs pre- and post-configuration tests, > Parrot::Configuration::Options::Test, has been revised to require an > up-to-date version of the 'prove' utility (from TAP::Parser 3.05 or > higher) so that options passed to Configure.pl on the command line can > be passed through to the test files in t/steps/. This is a problem too. Since Test::Harness 3.* changed the default output format, installing it has wider implications than I want to think about at the moment. Unless I am mistaken, the following lines added to Configure::Options::Test will cause Configure.pl to always fail unless the user has TAP::Harness installed. This means Test::Harness 3.* would now be required for everyone. +BEGIN { + eval { use TAP::Harness 3.05 (); }; + if ($@) { + print STDERR "[EMAIL PROTECTED] Must have modern 'prove' tests using Parrot::Configure::Options::Test\n"; + exit; + } +} > Please review. This patch is necessarily large and is likely to give > the 'patch' program fits. But you can play in a sandbox for the SVN > branch in which I developed this approach. To see > configuration-testing-in-parallel in action, call 'svn co > https://svn.perl.org/branches/tcif/' and run 'perl Configure.pl --test' > with whatever command-line options you customarily use when configuring > in trunk. Ok. I set up a fresh perl installation into which I could install TAP::Harness without messing up anything else, checked out a fresh copy of parrot, and applied your patch. Apart from some irrelevant issues with MANIFEST, MANIFEST.skip and some languages stuff, it all applied just fine. However, actually trying it out, I get *lots* and *lots* of errors on my Solaris system. I've attached the compressed output (uncompressed it's over 80k!) of Configure.pl --cc=gcc --link=gcc --ld=gcc --cxx=g++ --test The first major failure is here: t/steps/inter_progs-01.........................Can't call method "data" on an undefined value at (eval 20) line 7. # Looks like you planned 11 tests but only ran 2. # Looks like your test died just after 2. Dubious, test returned 255 (wstat 65280, 0xff00) Failed 9/11 subtests t/steps/inter_progs-02.........................Can't call method "data" on an undefined value at (eval 20) line 7. # Looks like you planned 11 tests but only ran 2. # Looks like your test died just after 2. Dubious, test returned 255 (wstat 65280, 0xff00) Failed 9/11 subtests [etc for the rest of inter_progs-??.t] Alas, I can't see how to narrow this down further, since I can't see how to use prove -v on individual tests that are expecting to somehow get the original Configure.pl command line, and the current output is unhelpful anyway. I tried adding print commands to the test, but all the problems are occurring inside a capture( sub {...} ) construct, so it's annoying to try to get information out in any useful way. $ prove --version TAP::Harness v3.09 and Perl v5.10.0 $ prove -v t/steps/inter_progs-01.t t/steps/inter_progs-01...... 1..11 ok 1 - use config::inter::progs; ok 2 - Parallel::Trace constructor succeeded not ok 3 - Step stored # Failed test 'Step stored' # at t/steps/inter_progs-01.t line 21. # got: '2' # expected: '1' ok 4 - inter::progs constructor returned defined value ok 5 - The object isa inter::progs ok 6 - inter::progs has description ok 7 - Tie::Filehandle::Preempt::Stdin->can('READLINE') ok 8 - The object isa Tie::Filehandle::Preempt::Stdin # Looks like you planned 11 tests but only ran 8. # Looks like you failed 1 test of 8 run. # Looks like your test died just after 8. Dubious, test returned 255 (wstat 65280, 0xff00) Failed 4/11 subtests Test Summary Report ------------------- t/steps/inter_progs-01.t (Wstat: 65280 Tests: 8 Failed: 1) Failed test: 3 Non-zero exit status: 255 Parse errors: Bad plan. You planned 11 tests but ran 8. Files=1, Tests=8, 3 wallclock secs ( 0.15 usr 0.01 sys + 1.39 cusr 0.22 csys = 1.77 CPU) Result: FAIL There are many, many, many other failures. Many are probably related to this failure to get the right compiler. There are also many strange warnings: t/steps/inter_make-01..........................Use of uninitialized value $make_c in substitution (s///) at config/inter/make.pm line 109. Use of uninitialized value $make_c in substitution (s///) at config/inter/make.pm line 109. ok t/steps/inter_make-02..........................Use of uninitialized value $make_c in substitution (s///) at config/inter/make.pm line 109. # Failed test 'Step stored; has previously been tested' # at t/steps/inter_make-02.t line 18. # got: '1' # expected: '2' # Looks like you failed 1 test of 9. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/9 subtests I don't know what's happening here. I don't know why it's complaining about an uninitialized variable. I also don't know what the complaint about 'Step stored' means. This was the first time run on a clean directory tree. I don't know what the got: '1' expected '2' actually means. I also can't figure out what's happening to the command line. Changing the command line to give a bogus C compiler doesn't seem to actually change anything: Configure.pl --cc=none --link=none --ld=none --cxx=none --test surely ought to run into problems, but doesn't seem any worse! Anyway, here is the test "Summary" report. Alas, with T::H: 3, it's not a terribly compact report, and there's no actual summary at the end, but here it is anyway. The full output is attached. Test Summary Report ------------------- t/steps/inter_progs-01.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 11 tests but ran 2. t/steps/inter_progs-02.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 11 tests but ran 2. t/steps/inter_progs-03.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 11 tests but ran 2. t/steps/inter_progs-04.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 10 tests but ran 2. t/steps/inter_make-02.t (Wstat: 256 Tests: 9 Failed: 1) Failed test: 3 Non-zero exit status: 1 t/steps/auto_gcc-01.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 9 tests but ran 2. t/steps/auto_gcc-02.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 9 tests but ran 2. t/steps/auto_gcc-03.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 10 tests but ran 2. t/steps/auto_gcc-04.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 12 tests but ran 2. t/steps/auto_gcc-05.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 10 tests but ran 2. t/steps/auto_gcc-06.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 12 tests but ran 2. t/steps/auto_gcc-07.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 13 tests but ran 2. t/steps/auto_gcc-08.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 12 tests but ran 2. t/steps/auto_gcc-09.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 13 tests but ran 2. t/steps/auto_gcc-10.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 11 tests but ran 2. t/steps/auto_gcc-11.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 11 tests but ran 2. t/steps/auto_gcc-12.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 12 tests but ran 2. t/steps/auto_gcc-13.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 13 tests but ran 2. t/steps/auto_backtrace-02.t (Wstat: 256 Tests: 12 Failed: 1) Failed test: 3 Non-zero exit status: 1 t/steps/auto_msvc-01.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 8 tests but ran 2. t/steps/auto_msvc-02.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 10 tests but ran 2. t/steps/auto_msvc-03.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 11 tests but ran 2. t/steps/auto_msvc-04.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 24 tests but ran 2. t/steps/auto_warnings-02.t (Wstat: 256 Tests: 9 Failed: 1) Failed test: 3 Non-zero exit status: 1 t/steps/auto_alignptrs-01.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 9 tests but ran 2. t/steps/auto_alignptrs-02.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 9 tests but ran 2. t/steps/auto_alignptrs-03.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 9 tests but ran 2. t/steps/auto_alignptrs-04.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 10 tests but ran 2. t/steps/auto_alignptrs-05.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 8 tests but ran 2. t/steps/auto_alignptrs-06.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 8 tests but ran 2. t/steps/auto_alignptrs-07.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 10 tests but ran 2. t/steps/auto_headers-02.t (Wstat: 256 Tests: 10 Failed: 1) Failed test: 3 Non-zero exit status: 1 t/steps/auto_sizes-01.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 9 tests but ran 2. t/steps/auto_sizes-02.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 29 tests but ran 2. t/steps/auto_byteorder-01.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 11 tests but ran 2. t/steps/auto_byteorder-02.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 15 tests but ran 2. t/steps/auto_byteorder-03.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 8 tests but ran 2. t/steps/auto_va_ptr-01.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 4 tests but ran 2. t/steps/auto_format-01.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 13 tests but ran 2. t/steps/auto_format-02.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 12 tests but ran 2. t/steps/auto_isreg-02.t (Wstat: 256 Tests: 10 Failed: 1) Failed test: 3 Non-zero exit status: 1 t/steps/auto_isreg-03.t (Wstat: 256 Tests: 10 Failed: 1) Failed test: 3 Non-zero exit status: 1 t/steps/auto_arch-02.t (Wstat: 256 Tests: 9 Failed: 1) Failed test: 3 Non-zero exit status: 1 t/steps/auto_arch-03.t (Wstat: 256 Tests: 11 Failed: 1) Failed test: 3 Non-zero exit status: 1 t/steps/auto_snprintf-01.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 7 tests but ran 2. t/steps/auto_snprintf-02.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 12 tests but ran 2. t/steps/auto_snprintf-03.t (Wstat: 65280 Tests: 2 Failed: 0) Non-zero exit status: 255 Parse errors: Bad plan. You planned 9 tests but ran 2. t/steps/auto_perldoc-02.t (Wstat: 256 Tests: 22 Failed: 1) Failed test: 3 Non-zero exit status: 1 Files=261, Tests=2680, 448 wallclock secs (12.22 usr 3.23 sys + 349.94 cusr 57.21 csys = 422.60 CPU) Result: FAIL Pre-configuration tests did not complete successfully; Configure.pl will not continue. at lib/Parrot/Configure/Options/Test.pm line 167. -- Andy Dougherty [EMAIL PROTECTED]
configure-test.out.bz2
Description: Output of perl Configure.pl --test