On Sat, 23 Feb 2008, James Keenan via RT wrote:

> The patch attached, diff.trunk.tcif.txt, addresses both RT 47391
> (t/configure/1*.t tests incorrectly rely on init::defaults_ and RT 47503
> (Remove config::init::defaults From configure tests).  It accomplishes
> the following:

Thanks for working on this.  Alas, it didn't work.  I don't have any more
time to debug this any further, but I'll share some observations I had:

> * In each test, the configuration step is first run just as it will be
> by Configure.pl.  The results of that step are recorded on disk.  The
> test file then reads from the disk the results of all previous
> configuration steps and uses that in tests for the step currently being
> tested.  Thus, for the testing of a given configuration step, individual
> tests have available to them all the information found in the
> Parrot::Configure object at that same point during regular configuration.

This sounds to me as if it assumes all the tests will be running in order 
every time.  For example, suppose I have a test failure with test number 
200.  Does that mean I have to rerun all 199 preceding tests every time I 
wish to change something in test 200?  Conversely, suppose all 200 tests 
run, but I want to go back and investigate a failure in test 10.  The 
stored results will reflect all 200 tests, so they won't be the same the 
next time I run test 10.  Will that be an issue?

More generally: Where is this cache stored?  How do we know if it's
still valid or not?  What invalidates this cache?

> * The harness which runs pre- and post-configuration tests,
> Parrot::Configuration::Options::Test, has been revised to require an
> up-to-date version of the 'prove' utility (from TAP::Parser 3.05 or
> higher) so that options passed to Configure.pl on the command line can
> be passed through to the test files in t/steps/.

This is a problem too.  Since Test::Harness 3.* changed the default 
output format, installing it has wider implications than I want to think 
about at the moment.  Unless I am mistaken, the following lines added to
Configure::Options::Test will cause Configure.pl to always fail unless
the user has TAP::Harness installed.  This means Test::Harness 3.* would
now be required for everyone.

    +BEGIN {
    +    eval { use TAP::Harness 3.05 (); };
    +    if ($@) {
    +        print STDERR "[EMAIL PROTECTED]  Must have modern 'prove' tests 
using Parrot::Configure::Options::Test\n";
    +        exit;
    +    }
    +}

> Please review.  This patch is necessarily large and is likely to give
> the 'patch' program fits.  But you can play in a sandbox for the SVN
> branch in which I developed this approach.  To see
> configuration-testing-in-parallel in action, call 'svn co
> https://svn.perl.org/branches/tcif/' and run 'perl Configure.pl --test'
> with whatever command-line options you customarily use when configuring
> in trunk.

Ok.  I set up a fresh perl installation into which I could install
TAP::Harness without messing up anything else, checked out a fresh copy
of parrot, and applied your patch.  Apart from some irrelevant issues
with MANIFEST, MANIFEST.skip and some languages stuff, it all applied
just fine.

However, actually trying it out, I get *lots* and *lots*  of errors on
my Solaris system.  I've attached the compressed output (uncompressed
it's over 80k!) of

    Configure.pl --cc=gcc --link=gcc --ld=gcc --cxx=g++ --test 

The first major failure is here:

t/steps/inter_progs-01.........................Can't call method "data" on an 
undefined value at (eval 20) line 7.
# Looks like you planned 11 tests but only ran 2.
# Looks like your test died just after 2.
 Dubious, test returned 255 (wstat 65280, 0xff00)
 Failed 9/11 subtests 
t/steps/inter_progs-02.........................Can't call method "data" on an 
undefined value at (eval 20) line 7.
# Looks like you planned 11 tests but only ran 2.
# Looks like your test died just after 2.
 Dubious, test returned 255 (wstat 65280, 0xff00)
 Failed 9/11 subtests 

[etc for the rest of inter_progs-??.t]

Alas, I can't see how to narrow this down further, since I can't see how
to use prove -v on individual tests that are expecting to somehow get
the original Configure.pl command line, and the current output is
unhelpful anyway.  I tried adding print commands to the test, but all
the problems are occurring inside a 
        capture( sub {...} )
construct, so it's annoying to try to get information out in any useful
way.

    $ prove --version
    TAP::Harness v3.09 and Perl v5.10.0
    $ prove -v t/steps/inter_progs-01.t
    t/steps/inter_progs-01......
    1..11
    ok 1 - use config::inter::progs;
    ok 2 - Parallel::Trace constructor succeeded
    not ok 3 - Step stored

    #   Failed test 'Step stored'
    #   at t/steps/inter_progs-01.t line 21.
    #          got: '2'
    #     expected: '1'
    ok 4 - inter::progs constructor returned defined value
    ok 5 - The object isa inter::progs
    ok 6 - inter::progs has description
    ok 7 - Tie::Filehandle::Preempt::Stdin->can('READLINE')
    ok 8 - The object isa Tie::Filehandle::Preempt::Stdin
    # Looks like you planned 11 tests but only ran 8.
    # Looks like you failed 1 test of 8 run.
    # Looks like your test died just after 8.
     Dubious, test returned 255 (wstat 65280, 0xff00)
     Failed 4/11 subtests 

    Test Summary Report
    -------------------
    t/steps/inter_progs-01.t (Wstat: 65280 Tests: 8 Failed: 1)
      Failed test:  3
      Non-zero exit status: 255
      Parse errors: Bad plan.  You planned 11 tests but ran 8.
    Files=1, Tests=8,  3 wallclock secs ( 0.15 usr  0.01 sys +  1.39 cusr  0.22 
csys =  1.77 CPU)
    Result: FAIL


There are many, many, many other failures.  Many are probably related to
this failure to get the right compiler.

There are also many strange warnings:

t/steps/inter_make-01..........................Use of uninitialized value 
$make_c in substitution (s///) at config/inter/make.pm line 109.
Use of uninitialized value $make_c in substitution (s///) at 
config/inter/make.pm line 109.
ok
t/steps/inter_make-02..........................Use of uninitialized value 
$make_c in substitution (s///) at config/inter/make.pm line 109.

#   Failed test 'Step stored; has previously been tested'
#   at t/steps/inter_make-02.t line 18.
#          got: '1'
#     expected: '2'
# Looks like you failed 1 test of 9.
 Dubious, test returned 1 (wstat 256, 0x100)
 Failed 1/9 subtests 

I don't know what's happening here.  I don't know why it's complaining 
about an uninitialized variable.  I also don't know what the complaint 
about 'Step stored' means.  This was the first time run on a clean 
directory tree.  I don't know what the got: '1' expected '2' actually 
means.

I also can't figure out what's happening to the command line.  Changing
the command line to give a bogus C compiler doesn't seem to actually
change anything:

    Configure.pl --cc=none --link=none --ld=none --cxx=none --test 

surely ought to run into problems, but doesn't seem any worse!

Anyway, here is the test "Summary" report.  Alas, with T::H: 3, it's not
a terribly compact report, and there's no actual summary at the end, but
here it is anyway.  The full output is attached.

Test Summary Report
-------------------
t/steps/inter_progs-01.t                   (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 11 tests but ran 2.
t/steps/inter_progs-02.t                   (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 11 tests but ran 2.
t/steps/inter_progs-03.t                   (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 11 tests but ran 2.
t/steps/inter_progs-04.t                   (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 10 tests but ran 2.
t/steps/inter_make-02.t                    (Wstat: 256 Tests: 9 Failed: 1)
  Failed test:  3
  Non-zero exit status: 1
t/steps/auto_gcc-01.t                      (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 9 tests but ran 2.
t/steps/auto_gcc-02.t                      (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 9 tests but ran 2.
t/steps/auto_gcc-03.t                      (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 10 tests but ran 2.
t/steps/auto_gcc-04.t                      (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 12 tests but ran 2.
t/steps/auto_gcc-05.t                      (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 10 tests but ran 2.
t/steps/auto_gcc-06.t                      (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 12 tests but ran 2.
t/steps/auto_gcc-07.t                      (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 13 tests but ran 2.
t/steps/auto_gcc-08.t                      (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 12 tests but ran 2.
t/steps/auto_gcc-09.t                      (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 13 tests but ran 2.
t/steps/auto_gcc-10.t                      (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 11 tests but ran 2.
t/steps/auto_gcc-11.t                      (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 11 tests but ran 2.
t/steps/auto_gcc-12.t                      (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 12 tests but ran 2.
t/steps/auto_gcc-13.t                      (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 13 tests but ran 2.
t/steps/auto_backtrace-02.t                (Wstat: 256 Tests: 12 Failed: 1)
  Failed test:  3
  Non-zero exit status: 1
t/steps/auto_msvc-01.t                     (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 8 tests but ran 2.
t/steps/auto_msvc-02.t                     (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 10 tests but ran 2.
t/steps/auto_msvc-03.t                     (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 11 tests but ran 2.
t/steps/auto_msvc-04.t                     (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 24 tests but ran 2.
t/steps/auto_warnings-02.t                 (Wstat: 256 Tests: 9 Failed: 1)
  Failed test:  3
  Non-zero exit status: 1
t/steps/auto_alignptrs-01.t                (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 9 tests but ran 2.
t/steps/auto_alignptrs-02.t                (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 9 tests but ran 2.
t/steps/auto_alignptrs-03.t                (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 9 tests but ran 2.
t/steps/auto_alignptrs-04.t                (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 10 tests but ran 2.
t/steps/auto_alignptrs-05.t                (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 8 tests but ran 2.
t/steps/auto_alignptrs-06.t                (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 8 tests but ran 2.
t/steps/auto_alignptrs-07.t                (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 10 tests but ran 2.
t/steps/auto_headers-02.t                  (Wstat: 256 Tests: 10 Failed: 1)
  Failed test:  3
  Non-zero exit status: 1
t/steps/auto_sizes-01.t                    (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 9 tests but ran 2.
t/steps/auto_sizes-02.t                    (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 29 tests but ran 2.
t/steps/auto_byteorder-01.t                (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 11 tests but ran 2.
t/steps/auto_byteorder-02.t                (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 15 tests but ran 2.
t/steps/auto_byteorder-03.t                (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 8 tests but ran 2.
t/steps/auto_va_ptr-01.t                   (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 4 tests but ran 2.
t/steps/auto_format-01.t                   (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 13 tests but ran 2.
t/steps/auto_format-02.t                   (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 12 tests but ran 2.
t/steps/auto_isreg-02.t                    (Wstat: 256 Tests: 10 Failed: 1)
  Failed test:  3
  Non-zero exit status: 1
t/steps/auto_isreg-03.t                    (Wstat: 256 Tests: 10 Failed: 1)
  Failed test:  3
  Non-zero exit status: 1
t/steps/auto_arch-02.t                     (Wstat: 256 Tests: 9 Failed: 1)
  Failed test:  3
  Non-zero exit status: 1
t/steps/auto_arch-03.t                     (Wstat: 256 Tests: 11 Failed: 1)
  Failed test:  3
  Non-zero exit status: 1
t/steps/auto_snprintf-01.t                 (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 7 tests but ran 2.
t/steps/auto_snprintf-02.t                 (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 12 tests but ran 2.
t/steps/auto_snprintf-03.t                 (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 9 tests but ran 2.
t/steps/auto_perldoc-02.t                  (Wstat: 256 Tests: 22 Failed: 1)
  Failed test:  3
  Non-zero exit status: 1
Files=261, Tests=2680, 448 wallclock secs (12.22 usr  3.23 sys + 349.94 cusr 
57.21 csys = 422.60 CPU)
Result: FAIL
Pre-configuration tests did not complete successfully; Configure.pl will not 
continue. at lib/Parrot/Configure/Options/Test.pm line 167.

-- 
    Andy Dougherty              [EMAIL PROTECTED]

Attachment: configure-test.out.bz2
Description: Output of perl Configure.pl --test

Reply via email to