Note that we're going off topic from the original goal, and I just want to re-iterate that I fully support smaller, incremental changes. But since I like talking about lit so much, I can't help but chime in :)
If we *did* want to move to a lit based system for the end to end based tests, the first step would be to make an LLDBTestFormat and teach it to literally just call dotest.py in single-process mode with the path to a specific test file. That would be a *great* first step, and also very manageable and bite sized. I think we would see a measurable reduction in flakiness just from this change. On the other side though, I still think it is very important to increase test coverage of *non* end-to-end tests. When you increase the granularity of the tests you're able to write, a lot more things become possible. For non end-to-end based tests, we can still use a more traditional llvm style lit test based on lldb-test (which is still new, but the foundation is there). On Wed, Jan 17, 2018 at 2:59 PM Vedant Kumar <v...@apple.com> wrote: > Lit provides some helpful utilities which make it easier to write portable > tests. E.g %t, for temporary test directories, and portable replacements > for things like `diff -r`. This is how compiler-rt's end-to-end tests are > structured, and we haven't needed any build-system like functionality there. > > vedant > > > On Jan 17, 2018, at 2:56 PM, Jim Ingham <jing...@apple.com> wrote: > > > > > >> On Jan 17, 2018, at 2:55 PM, Adrian Prantl <apra...@apple.com> wrote: > >> > >> > >> > >>> On Jan 17, 2018, at 2:50 PM, Jim Ingham <jing...@apple.com> wrote: > >>> > >>> > >>> > >>>> On Jan 17, 2018, at 2:49 PM, Adrian Prantl <apra...@apple.com> wrote: > >>>> > >>>> > >>>> > >>>>> On Jan 17, 2018, at 2:31 PM, Zachary Turner <ztur...@google.com> > wrote: > >>>>> > >>>>> I don't think new test authors really need to add CMake any more so > than they currently need to understand Make. Which is to say, not very > much. Most Makefiles are currently 1-2 lines of code that simply does > nothing other than include the common Makefile. > >>>>> > >>>>> On the other hand, CMake defines a lot of constructs designed to > support portable builds, so actually writing and maintaining that common > CMake build file would be much easier. The existing Makefile-based system > already doesn't require you to understand the specific compiler invocations > you want. Here's 3 random Makefiles, which is hopefully representative > given that I pulled them completely at random. > >>>>> > >>>>> breakpoint-commands/Makefile: > >>>>> LEVEL = ../../../make > >>>>> CXX_SOURCES := nested.cpp > >>>>> include $(LEVEL)/Makefile.rules > >>>>> > >>>>> functionalities/inferior-assert: > >>>>> LEVEL = ../../make > >>>>> C_SOURCES := main.c > >>>>> include $(LEVEL)/Makefile.rules > >>>>> > >>>>> > >>>>> types: > >>>>> LEVEL = ../make > >>>>> # Example: > >>>>> # > >>>>> # CXX_SOURCES := int.cpp > >>>>> include $(LEVEL)/Makefile.rules > >>>>> > >>>>> None of this is particularly interesting. There are a very few > tests that need to do something weird. I opened 10 other random Makefiles > and still didn't find any. I don't believe it would be hard to support > those cases. > >>>>> > >>>>> So now instead of "understand Make" it becomes "understand CMake". > Whic is already a requirement of building LLVM. > >>>> > >>>> Fair point. I would suggest that I'll try to make LLDB's testsuite > build out-of-tree using the existing Makefile system. That should be a > generally useful first step. After doing this I will hopefully have a much > better understanding of the requirements of the Makefiles and then we can > revisit this idea with me actually knowing what I'm talking about :-) > >>>> > >>>>> If our test suite was lit-based where you actually had to write > compiler invocations into the test files, I would feel differently, but > that isn't what we have today. We have something that is almost a direct > mapping to using CMake. > >>>> > >>>> Question: how would you feel about converting the Makefiles to > LIT-style .test files with very explicit RUN-lines? > >>> > >>> I'm not sure what you mean by this. > >> > >> Instead of using a build system at all to build the tests, we would > hard-code the compiler and linker invocations without encoding any > dependencies. Because we still need this to be configurable, it would > probably look something like this: > >> > >> RUN: %CXX test.cpp -O0 %CXXFLAGS -o test.exe > >> RUN: %test_driver test.exe mytest.py > > > > I'm worried we'd back into building another make system over time. What > advantage would we get from this. > > > > Jim > > > > > > > >> > >> -- adrian > >> > >>> > >>> Jim > >>> > >>> > >>>> > >>>> -- adrian > > > >
_______________________________________________ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev