On Mon Feb 25 13:09:04 2008, doughera wrote: > > This sounds to me as if it assumes all the tests will be running in > order > every time.
For the most part, this is true. However, once we work out the kinks, this will not be cause for alarm. We want our tests to simulate the development of the Parrot::Configure object over the course of the configuration steps. Currently, the configuration step tests do that but very imperfectly. That's because as a kludge for having the information from steps 1..(N-1) available, I had the individual test files run step 2, init::defaults, and, in certain cases, some other steps as well. You and chromatic have argued that that approach is ultimately inadequate, and I concur. To remedy this situation, however, we have to test in configuration step order and devise a way for configuration step N, once it has succeeded, to pass along information about the Parrot::Configure object to step N+1. That means we have to let go of the notion that a test of a particular configuration step can be completely self-contained. This takes some getting used to, because most of us who have come up in the Perl testing culture over the past decade have implicitly assumed that all test scripts ought to be completely self-contained programs under all circumstances. The circumstances here, I argue, are different, so the approach is likely to be different. In this new approach, in the first test of a given configuration step (e.g., t/steps/init_manifest_01.t for init::manifest), Parrot::Configure::Parallel::Trace internally runs config::init::manifest::runstep(): my $trace = Parrot::Configure::Parallel::Trace->new($0); ok(defined $trace, "Parallel::Trace constructor succeeded"); is($trace->store_this_step(), 1, "Step stored"); ... and records the results on disk in a Storable file, ~/.configure_parallel.sto, found under your home directory. Because it's recording the results of runstep mainly for the use of the *next* configuration step, it only needs to do this once. So this action is not repeated in, say, t/steps/init_manifest_02.t. The test file continues onward. Once it has constructed a new Parrot::Configure object, the guts of that object are ripped out and replaced with the results of configuration at step N-1. my $conf = Parrot::Configure->new; $conf->refresh($trace->get_previous_state()); So when the test file comes to '$step->runstep($conf)', it's passing to the current step's runstep method all the knowledge about configuration that was accumulated in all the preceding steps. The cache file is deleted by Parrot::Configure::Options::Test::new(), which is called near the beginning of Configure.pl. This method call also puts the configuration step tests in the proper order. But the tests are only run and the cache only rebuilt if the user calls '--test' (or '--test=configure') on the command-line. Most users don't do that. Most users currently only see the configuration tests (not just the step tests, but the framework tests in t/configure/ and the post-configuration tests in t/postconfigure) when they call 'make test'. My proposal pulls these tests out of 'make test', thereby speeding up 'make test' considerably. Average users will also not be required to upgrade to Test-Harness-3.05 or greater. If, in the new approach, you want to run a particular test file with 'prove', you can do so easily under most circumstances. If you have already successfully completed configuration (e.g., 'perl Configure.pl --test=configure') and wish to hack on a particular config step class and its associated tests, then you leave the cache file on disk, do your revisions, and call 'prove' on that particular step. The test file will consult the cache for the results of all previous steps and refresh that file's new Parrot::Configure object with the results of those steps. The one approach where this won't work is when particular tests fail, which they will do so largely for the same reasons they fail right now, namely, that I haven't written the best possible test file (where 'best possible test file' is measured by: Did it pass on all of Andy's boxes? :-) ). This is what has happened in your case. But the fact that the tests did *not* begin failing until step 5, inter::progs -- where my tests have failed on your boxes previously -- suggests to me that the basic approach is workable. > This means Test::Harness 3.* > would > now be required for everyone. > I suspect that at some point in the future, we will want to make use of Test::Harness 3.* more broadly. But here I'm proposing only to use it -- more specifically, >= 3.05 -- to get the latest version of 'prove', which enables us to pass command-line options on to tests (and which was specifically commissioned for this approach). So only the subset of users who hack on the configuration system need upgrade at this time. > > Configure.pl --cc=gcc --link=gcc --ld=gcc --cxx=g++ --test > > The first major failure is here: > > t/steps/inter_progs-01.........................Can't call method > "data" on an undefined value at (eval 20) line 7. > # Looks like you planned 11 tests but only ran 2. > # Looks like your test died just after 2. > Dubious, test returned 255 (wstat 65280, 0xff00) > Failed 9/11 subtests As suggested above, my hunch is that if we can get this step to pass, everything after it will pass as well. Can I assume that, with the exception of '--test', the options you call above would normally result in a successful configuration? Thanks for taking the time to examine this. kid51
