Gary King wrote: >> Part of the problem with test-op is that the desired behavior has not >> been specified by the ASDF community. Because of the nature of ASDF, it >> is impossible for >> >> (asdf:test-system <system>) >> >> to return a value indicating whether or not <system> has passed its >> tests. > > FWIW, I recently changed operate-on-system (and therefore > asdf:test-system, asdf:load-system, etc.) to return the operation > instance that they processed. > > If ASDF also added and documented (shocking!) a few canonical slots in > the test-op class then these could be used to transmit information. As a > first pass, I'd suggest > > test-properties - a property list of test system specific stuff > test-result - a keyword from the list :success, :failures, > :errors, errors-and-failures > >> Some time ago, I proposed that ASDF provide a stream argument to the >> test-op, providing a place into which the testing tool could dump its >> test report for human inspection. I can't say that this suggestion >> received universal approbation. Or disapprobation. It merely received >> near-universal lack of interest! ;-) > > Sigh. I think this is a good idea; if we added a test-stream slot to the > test-op, then test systems could find it there. > > A tricky part is that test systems may not want to rely on the existence > of ASDF while running. So they'll need some sort of adaptor that adds > stuff to the ASDF operation if it exists. Would it make sense to define > and export *current-operation* to make this easier?
You'd still need the asdf stuff to get at this. The alternative would be to define a special variable that gives direct access to the stream, which is not an asdf-dependent entity. The problem with that approach is that you're pretty much doomed to crap up the CL-USER namespace. > >> Related to the question of what the test-op should provide to its >> invoker is the question of how dependencies should be propagated. > > > I think that the default is that test-op depends on load-op. Right, but what about testing system <foo> --- do we test <foo>'s depended on systems? My guess is that the correct answer to this is "it depends." If you have a standard set of libraries, the answer will typically be "no." OTOH, if you have a very large application (e.g., a web site with many components providing different features), you may well want to test the subsystems that are part of the top-level system. > > other thoughts? We have found in practice that we end up creating a test-<foo> system definition to complement our <foo> system definitions. The reason for this is that the existing ASDF structures have only components that are unconditional. For the test op we typically want to load new files. A second item is that the test system typically :depends-on the test framework library. This suggests a question: is there a nice way to specify additional files that need to be loaded ONLY for the test-op? I can think of some ways to achieve this with the existing ASDF framework, but they would be cumbersome and non-standard. Would it be possible to simply add a :test-components argument to defsystem? Note that the use of additional <test>-foo systems also helps us tailor subcomponent testing. Another note --- I found in an earlier attempt to tweak the test-op that it worked poorly to subclass operations (as opposed to components). IIRC, this was because of the way dependent operation instances are built. Alas, I didn't take good notes when I did this, and have garbage-collected much of what I learned. r _______________________________________________ asdf-devel mailing list asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel