On 1 December 2015 at 19:09, David Blaikie <dblai...@gmail.com> wrote: > Not sure I follow - I'm not suggesting adding objdump/instruction checking > to existing large programs in the test-suite, but adding other tests to the > test-suite that do this and have appropriate input for it to be a > reliable/useful form of testing.
Sorry, I was building to that... But currently, the test-suite compiles the code into an executable, then run it. Changing that premise means we'll have something other than the test-suite we always had, and may bring other problems, like too many hats for the infrastructure to keep, especially now that James is moving it to integrate with lit. We may decide this is the way to go, but we need to make clear all the pros and cons of doing so beforehand. > This is perhaps an interesting part/point: In theory I would think that > these tests should be able to be phrased in a compiler-agnostic way, since > they're testing the compiler as a whole and testing behavior that the source > code demands, no? There's where things get interesting. Say we change the test-suite to instead of compiling and running, just compile to asm/obj and grep/objdump to match instructions, ELF sections, debug symbols, etc. Some tests are easy to make compiler independent. Intrinsics tests fall into that category. Most currently in Clang tests could move directly to the test-suite as soon as we have such infrastructure. Other tests, however, are specific to Clang+LLVM that may make no sense to GCC or ICC, ARMCC etc. Those tests, would have to be in yet another separate directory, that is only enabled if the compiler is Clang, assuming the back-end is always LLVM. These are all options that, for me, have equal value with keeping them in Clang. AFAICS, they stir the same kind of problems wherever they are tested. >> would account for most of those problems, but it would also put a >> stress on slow/expensive/experimental hardware support. It's a >> balance, I think. > > Why would it require slow/expensive/experimental hardware? (& if it does > require that, how do you expect average developers to run them on a regular > basis?) You misunderstood. It would "put a stress on", not "require". Basically, it would increase the cost of testing on those classes of hardware more than off-the-shelf hardware by a non-trivial amount. This goes back to our discussion on buildbots, where some targets are harder to find hardware than others, and when you do, they're slower. Meaning the more variations we need to test, the more hardware you have to have to make it in any reasonable time. While a single 64-core Xeon can cope with most testing (including self-hosting, test-suite, sanitizers, libc++) on a per-commit basis, there is nothing similar, widely available, today for most other targets. But that's another discussion entirely, let's not dwell into that again. :) cheers, --renato _______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits