Hi, FWIW, I'm happy with moving these tests into the test-suite. They do make sense to be expressed in a compiler-neutral manner, even though some may test clang-specific behaviour.
There is a UnitTests directory already, and adding the ability to run LIT tests should be simple within the new CMake framework. James On Tue, 1 Dec 2015 at 19:09, David Blaikie <dblai...@gmail.com> wrote: > On Tue, Dec 1, 2015 at 10:43 AM, Renato Golin <renato.go...@linaro.org> > wrote: > >> On 1 December 2015 at 18:29, David Blaikie <dblai...@gmail.com> wrote: >> > Are they things the test-suite couldn't (either technically or >> > philosophically) cover, or only that it doesn't cover it at the moment, >> but >> > could do so? >> >> IMO, it's a philosophical issue. The test-suite is a whole-program >> execution, and all that matters is the end result, the output of the >> program, not the program itself. >> >> Those tests are mostly large (> 3 lines of code) so, if we were trying >> to objdump and match for instruction patterns, any optimisation >> variance could destroy them, as well as you may find the pattern >> elsewhere in the same object, not belonging to the function you want >> (inlining, etc). > > > Not sure I follow - I'm not suggesting adding objdump/instruction checking > to existing large programs in the test-suite, but adding other tests to the > test-suite that do this and have appropriate input for it to be a > reliable/useful form of testing. > > >> All in all, all the problems with reducing C code >> would be pertinent here, so a big hammer to kill small issues. On the >> other hand, the Clang based asm/obj examples are all small, and >> hand-crafted to suit the problem being tested. >> >> We could, however, add a new directory in the test-suite for that kind >> of tests, like we already have for regressions in C/C++. > > > If a separate directory makes it more clear what the purpose is, sure, > seems fine to me, but I don't have strong or informed opinions on exactly > how the test-suite is laid out. > > >> That would >> dissociate the front-end from the back-end, and essentially any >> compiler used would have to abide by the rules of the test (which may >> not be true for some compilers). >> > > This is perhaps an interesting part/point: In theory I would think that > these tests should be able to be phrased in a compiler-agnostic way, since > they're testing the compiler as a whole and testing behavior that the > source code demands, no? > > >> I don't mind where they end up, really, but we need that kind of test >> somewhere. Clang is an easy place because we know we have it built >> when the Clang repo is checked out. >> >> >> > That would sort of defeat the point of having the testing and projects >> > separated though - it would tie the tests together and produce most of >> the >> > undesirable outcomes of having single end-to-end tests. >> >> Right, I agree, but since the relationship between Clang and LLVM is >> non-trivial, and there are lots of changes that need to be done on >> both sides, I can't see why we shouldn't have tests that span across >> projects. >> >> Clearly, the sanitizer tests need Clang and Compiler-RT. Some Libc++ >> tests need Compiler-RT, others need libunwind, on ARM I need Clang to >> build it. There are a lot of dependencies that are not there by >> accident, but by design. >> >> The more you move tests away from the big targets (LLVM, Clang), the >> less they're ran by people committing patches, and the harder will be >> to pick up the failures. Having said that, a decent buildbot coverage >> would account for most of those problems, but it would also put a >> stress on slow/expensive/experimental hardware support. It's a >> balance, I think. >> > > Why would it require slow/expensive/experimental hardware? (& if it does > require that, how do you expect average developers to run them on a regular > basis?) > > >> >> >> > (it would mean that at least the pure (or at least non-Clang) LLVM >> developer >> > would have a test to run where they would not if the test remained in >> Clang >> > only) >> >> That is a good point. Right now, RT/libc++ tests in that category are >> controlled by enabling/disabling support via CMake flags. We may be >> able to do the same with Clang (if LLVM is not built with it, disable >> asm/obj tests). Moving them all to a regression package in the >> test-suite would be another option, but one that would take a lot >> longer... >> >> cheers, >> --renato >> >
_______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits