Brad King wrote:
David Abrahams wrote:
* logfile scraping is too hopelessly fragile to make for a good testing
system, and there are better and possibly even easier alternatives.
The question here is whether one wants to test with the same tools users
might use to build the project. If one user's tool doesn't provide
per-rule information then we need log-scraping to test it. That doesn't
mean we can't test some tools without log-scraping though.
I don't quite get "That doesn't mean we can't test some tools without
log-scraping".
I see two different cases here. There's the developer working under visual
studio or emacs who wants to run some tests. This guy knows (or should know)
how to find what compile/link flags were used, chase down warnings and whatnot.
In this case the message "test X failed, go look at the logs" is fine. Let
the user use his tool.
Then there's the 'tester' who operates testing slave boxes that poll svn, build
and run things. The test slave operator doesn't care about which tool, just
that it works. The output of these slave runs ends up on a website somewhere,
and in this case the information about the build should be exhaustive: every
step, every flag, every warning, every command linee, every
pass/fail/warn/notrun/timeout should be available, so that the library author
can get as much info as possible w/o asking the slave operator to dig around
in his build.
Frankly I'm not sure what logfile scraping has to do with the
structural problems you've mentioned.
I'm only referring to the test part of the anti-logscraping code. The
python command wrappers are there to avoid log scraping, but if the
tests were run through CTest then no log scraping would be needed.
One problem with the current implementation is that every test is a toplevel
target; this is easy to fix, if you just write the tests to flatfiles in the
build hierarchy (like ctest does it) and use a driver script to find the files
and run the tests (that is nice and clean for tests that involve just running
compiled binaries, I haven't thought about how it would work for compile and
compile-fail tests). I have since converted a different project (as large as
boost) to this method and am very happy with the results. You can easily
customize the test-running business... its just a python script, which seems to
me the right tool for the job, a scripting task. Getting information about
building/testing out to python, intact and in its entirety, makes communicating
with servers a real pleasure (using xmlrpc under python, for instance, to talk
to the xmlrpc plugin inside a Trac instance).
If I remember correctly the 'log scraping' was more about ctest having to scrape
the build log to find compile errors. Is this no longer necessary?
* Boost developers need the ability to change something in their
libraries and then run a test that checks everything in Boost that
could have been affected by that change without rebuilding and
re-testing all of Boost (i.e. "incremental retesting").
How does the current solution solve that problem (either Boost.Build or
the current CMake system)?
The large number of high-level targets places many rules in the outer
make level which leads to very long startup times (look at
CMakeFiles/Makefile2, which make needs to parse many times).
Just curious: why does make need to parse the same file many times?
I haven't looked at it in detail recently, but on quick inspection I
think it is now just twice. Each of the two times works with separate
rules so they could probably be split into two files. However, the file
has never been very big for any of our projects because we don't have a
huge number of top-level targets (since VS doesn't work in that case).
Boost needs to build libraries, documentation, and other files to be
placed in the install tree. Rules to build these parts can fit in
relatively few high-level targets and should certainly use them.
Sorry, what "should certainly use" what? Rules should use the targets?
Bad wording on my part. I meant that it is fine to use top-level
targets to drive the build of libraries and documentation.
However, there are some disadvantages:
(a) If one test fails to compile none of its tests can run
(b) A bad test may accidentally link due to symbols from another test
c) Adding a feature to a library requires modifying existing test code.
I don't understand what you mean here. Are you saying that to test a
new feature, the test dispatcher needs to be updated to link in the new
test? FYI, CMake provides a command to generate the dispatcher for you
(create_test_sourcelist).
CTest doesn't do log-scraping
to detect errors. Every test gets run individually and its output is
recorded separately. Boost's current system puts the tests inside the
build and then jumps through hoops to avoid log-scraping of the
results.
What kind of hoops?
It runs all the tests through python command wrappers to capture
individual output, and therefore has to generate its own compiler
command line invocations instead of using CMake's knowledge of the
native tools. Currently -c and -o options are hard-coded AFAICS.
Right... do you see a different way to get this done? Again, the
thinking here was that:
1. This is really only necessary on build slaves, and those are either make or
nmake.
2. I wanted to be as close to the invocation of the compiler as possible, to
capture as much as possible (execution time, environment, arguments, exit
status, etc).
I suppose you could have cmake generate a target that first calls a python
script, passing the command line it is about to call and some information about
what it is doing ("i'm compiling foo.o from foo.cpp, the command line is
'blah'"), then execute the command, but then what do you do about the exit
status... (ah, I see you've suggested something better below)
This brings us to the log-scraping issue in general. CMake permits
users to build projects using their native tools, including Visual
Studio, Xcode and Makefiles. In order to make sure builds with these
tools work, the test system must drive builds through them too.
Testing with just one type of native tool is insufficient.
That's at least somewhat debatable. To get results for all the
different toolchains would require more testing resources, would it not?
Yes. In our model we ask users to contribute testing resources for the
toolchains they want supported which we don't have. If no one cares
enough about a platform/compiler to submit tests, we don't need to
support that platform.
Since the native tools do not support per-rule reporting log-scraping
is necessary.
Also somewhat debatable. If we can get xcode to invoke
"boost-g++-wrapper" instead of "g++," we can still get per-rule
reporting, right?
If per-rule reporting is not available from a native tool we have to do
log-scraping. What's debatable is whether we can work around a lack of
explicit support in the native tools.
We could make this a CMake feature by teaching the generators to wrap
the compiler up with a tool we distribute with CMake. Then you won't
have to hack the compilation rule variables for Boost or depend on python.
Presumably the name of this tool (let's call it CWrap?) would have some
interface that you could implement however you like... and if that is
implementable in python, you've given your users lots of ways to tweak their
build system. We're OK with being dependent on python.
Problem (a) is automatically handled by the testing solution I propose
above since test results are recorded and reported individually.
Sorry, what did you propose above?
Testing with ctest's --build-and-test feature. The entire build and
execution of every test would be captured independently of other tests.
Or a python script that does what ctest's --build-and-test does...
IMV a lot more flexibility, a lot less code.
Problem (c) is a lack of convenience when the build error is subtle
enough to require the compiler command line to diagnose it (which in
my experience is very rare).
Rare, but when you need it, you really need it.
Well, one could always reproduce the build locally or get help from the
person running the machine with the problem...hence "lack of convenience" :)
However, I think our discussion above concludes that log-scraping
avoidance is not the main problem. It can be made to work.
P.S. Boost's CMake code currently uses file(TO_NATIVE_PATH) but then
references the result variable in CMake code. This causes a build
tree with spaces in the path to create files in a directory with
escaped backslashes in its name. Also, will python deal with
non-escaped backslashes in windows paths inside the strings configured
into the scripts?
I can tell you lots about python, but I don't understand your question.
Could you rephrase?
The code in question tells CMake to generate a python script that looks
like this (on Windows):
sys.path.append("c:\path\with\backslashes\to\some\file.txt")
# ^^ escape sequence?
I'd have to look back. This stuff was indeed working on windows; anyhow, looks
like we can detangle a lot of this with some of your help making tweaks to cmake
itself.
-t
_______________________________________________
Boost-cmake mailing list
Boost-cmake@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-cmake
