dblaikie added a comment. In D107049#3103984 <https://reviews.llvm.org/D107049#3103984>, @lhames wrote:
> In D107049#3101456 <https://reviews.llvm.org/D107049#3101456>, @dblaikie > wrote: > >> In D107049#3100630 <https://reviews.llvm.org/D107049#3100630>, @rnk wrote: >> >>> So, to back up a bit, do I understand correctly that this change adds tests >>> to the check-clang test suite that JIT compiles C++ code for the host and >>> throws C++ exceptions? Can we reconsider that? >>> >>> We have a policy of not running execution tests in the clang test suite >>> because we know they always turn out to be unreliable, flaky, and highly >>> dependent on the environment rather than the code under test. Integration >>> tests are great, everyone needs them, but they should definitely not be >>> part of the default set of tests that developers run anywhere and >>> everywhere and expect to work out of the box, regardless of the >>> environment. +@dblaikie to confirm if I am off base here about our testing >>> strategy, and maybe Lang can advise us about past JIT testing strategies. >> >> Yeah, seems we've had some precedent for this for a while, as @lhames says >> (doing some archaeology I see one of the clang-interpreter tests added here, >> for instance: >> https://github.com/llvm/llvm-project/commit/1d487617f20ab9df65ab60d6cf9431ef288312ab > > Also all of the MJCIT and Orc tests in LLVM. I think @rnk was asking/drawing a distinction between execution tests in LLVM (which is at least only testing LLVM code in LLVM) and execution tests in Clang (which is both execution, and cross-project) - so I was just talking about the Clang->LLVM->execution tests. Though I think in LLVM there'd also be some opportunity to find some ways to isolate/test JIT infrastructure with less execution (had sort of hoped in the early days of Orc that the layered architecture would allow more isolated/unit testing to robustly test pieces without execution for that testing). >> I push back pretty hard on end-to-end tests in clang in most cases - one >> place that slips through either untested or with end to end tests is >> anything in MCOptions, etc, since they aren't in IR and are only observable >> through their effect on the resulting assembly. And JITs - I didn't actually >> know we had live JIT tests up in clang - and yeah, would rather we didn't - >> perhaps via a mode in such frontends that emits the IR at various points, >> but it'll be a pretty significant investment to make something fairly >> robust, I would imagine (sort of like lldb reproducers replays (which are >> being removed due to unmaintainability, by the sounds of it) but being able >> to stub out the actual execution) - hmm, perhaps a remote execution mock >> would be feasible? That'd at least remove the execution part, make it >> portable to run on non-native environments - but wouldn't remove the >> end-to-end-ness, for that maybe you'd need a mock target that worked as >> though it ran IR natively, perhaps? Not sure. > > The LLVM MCJIT and Orc tests have been stable enough once we constrained the > environments that they run in. I don't see why this would be any less stable. > The features being exercised (essentially the object runtimes) are likely to > be more stable in practice than the environments being tested in the LLDB > tests. > >> Then maybe the more end-to-end-y tests could move out to cross-project-tests. > > Are you suggesting execution tests for cross-project-tests? I think cross-project-tests (originally debuginfo-tests) has always had/been a place for execution tests. Not all of them, and still value in having lit-style non-execution cross-project tests where possible for more robust/variation testing where possible, with fewer full end-to-end-including-execution tests where possible (always going to be fuzzy lines of what's worth the cost/benefit, etc - and we're all going to have different points on that spectrum). CHANGES SINCE LAST ACTION https://reviews.llvm.org/D107049/new/ https://reviews.llvm.org/D107049 _______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
