On 1 December 2015 at 18:29, David Blaikie <dblai...@gmail.com> wrote: > Are they things the test-suite couldn't (either technically or > philosophically) cover, or only that it doesn't cover it at the moment, but > could do so?
IMO, it's a philosophical issue. The test-suite is a whole-program execution, and all that matters is the end result, the output of the program, not the program itself. Those tests are mostly large (> 3 lines of code) so, if we were trying to objdump and match for instruction patterns, any optimisation variance could destroy them, as well as you may find the pattern elsewhere in the same object, not belonging to the function you want (inlining, etc). All in all, all the problems with reducing C code would be pertinent here, so a big hammer to kill small issues. On the other hand, the Clang based asm/obj examples are all small, and hand-crafted to suit the problem being tested. We could, however, add a new directory in the test-suite for that kind of tests, like we already have for regressions in C/C++. That would dissociate the front-end from the back-end, and essentially any compiler used would have to abide by the rules of the test (which may not be true for some compilers). I don't mind where they end up, really, but we need that kind of test somewhere. Clang is an easy place because we know we have it built when the Clang repo is checked out. > That would sort of defeat the point of having the testing and projects > separated though - it would tie the tests together and produce most of the > undesirable outcomes of having single end-to-end tests. Right, I agree, but since the relationship between Clang and LLVM is non-trivial, and there are lots of changes that need to be done on both sides, I can't see why we shouldn't have tests that span across projects. Clearly, the sanitizer tests need Clang and Compiler-RT. Some Libc++ tests need Compiler-RT, others need libunwind, on ARM I need Clang to build it. There are a lot of dependencies that are not there by accident, but by design. The more you move tests away from the big targets (LLVM, Clang), the less they're ran by people committing patches, and the harder will be to pick up the failures. Having said that, a decent buildbot coverage would account for most of those problems, but it would also put a stress on slow/expensive/experimental hardware support. It's a balance, I think. > (it would mean that at least the pure (or at least non-Clang) LLVM developer > would have a test to run where they would not if the test remained in Clang > only) That is a good point. Right now, RT/libc++ tests in that category are controlled by enabling/disabling support via CMake flags. We may be able to do the same with Clang (if LLVM is not built with it, disable asm/obj tests). Moving them all to a regression package in the test-suite would be another option, but one that would take a lot longer... cheers, --renato _______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits