On Fri, Aug 31, 2012 at 1:38 PM, Chandler Carruth <[email protected]>wrote:
> On Fri, Aug 31, 2012 at 1:31 PM, Daniel Dunbar <[email protected]> wrote: > >> >> >> On Friday, August 31, 2012, Chandler Carruth wrote: >> >>> On Fri, Aug 31, 2012 at 12:33 PM, Daniel Dunbar <[email protected]>wrote: >>> >>>> On Fri, Aug 31, 2012 at 12:23 PM, David Blaikie <[email protected]> >>>> wrote: >>>> > On Fri, Aug 31, 2012 at 11:53 AM, Daniel Dunbar <[email protected]> >>>> wrote: >>>> >> On Fri, Aug 31, 2012 at 11:23 AM, David Blaikie <[email protected]> >>>> wrote: >>>> >>> Daniel (& anyone else), >>>> >>> >>>> >>> Do you know if the documentation here ( >>>> >>> http://llvm.org/docs/TestingGuide.html#rtcustom / >>>> >>> >>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/TestingGuide.html?view=diff&r1=36059&r2=36060 >>>> >>> ) is still valid? It doesn't appear to be, since my change below >>>> >>> doesn't do particularly interesting quoting/escaping gymnastics & >>>> >>> seems to work fine. >>>> >>> >>>> >>> Any ideas? Should we update the documentation? Does anyone know >>>> >>> what/when/how/why this changed? >>>> >> >>>> >> What is actually going on here is that the LLVM test suite and the >>>> >> Clang test suite use a different style of test. >>>> >> >>>> >> The LLVM test suite is technically written using Tcl syntax, and the >>>> >> Clang test suite uses shell syntax. The document in question is >>>> >> referring to the LLVM test suite. >>>> >> >>>> >> The fact that the test suites use different styles is an unfortunate >>>> >> and poorly documented historical accident. Ideally we would kill off >>>> >> the Tcl style and only use the shell style. >>>> > >>>> > I see - I think I'd heard some of that in various discussions. Where >>>> > is the option for shell V tcl specified? >>>> >>>> The top level lit.cfg specifies the "test format", e.g.: >>>> config.test_format = lit.formats.ShTest(execute_external) >>>> >>>> Actually, apparently I lied the LLVM suite has moved over to the Sh >>>> style tests and I never noticed: >>>> http://llvm.org/viewvc/llvm-project?view=rev&revision=159525 >>>> >>>> So in fact the document in question is completely out of date. >>>> >>> >>> Sorry for missing the documentation... >>> >>> >>>> >>>> > >>>> >> Also, what is a "lesser lit-like" test runner? >>>> > >>>> > In this case, it's a shell script that attempts to transform RUN lines >>>> > into a shell script to run a lit test - it works for the basic cases. >>>> >>>> Dare I ask why you would do this instead of just running lit? >>> >>> >>> We run the tests in a distributed fashion, one test per worker, and the >>> start-up overhead of python combined with the lit stuff takes more time >>> than most of the test cases by a factor of 10 to 1000. ;] The shell script >>> starts up very very fast. >>> >> >> Makes sense. >> >> What if lit had a mode to generate all the scripts and then you just >> farmed them out? That would avoid having to duplicate any of the script >> parsing code. >> > > This would work, although it wouldn't be appreciably better than our > current arrangement imo. > > My long term idea is I would like to split lit into two pieces. One for > running a test case, and one for managing the suite of tests. > > The former would be well suited to a very efficient C++ program. I would > teach it just enough shell-like syntax to cope with the existing test > suite, possibly simplifying the shell used in the existing test suite as > necessary. This would then remove the dependency on bash (which hurts > windows a lot and makes it confusing as the bash versions change), make > the test suite run a small amount faster, and solve our specific problem > internally. > > The latter I'd like to eventually rewrite in C++ as well, specifically as > part of Ninja, or a plugin, or at least re-using its code to do really > efficient process management. Ninja is able to keep my 16 cpus between 20% > and 50% busier than lit is currently, and some experiments show that as the > # of cpus goes up, this is only going to get worse. > > Of course, none of this is "short term". ;] > For what it's worth, I wrote a lit-like testrunner in go a while ago. It's pretty hacky but was able to pass all clang tests. It wasn't all that much faster than lit (maybe 20%?), and the limiting factor seemed to be launching processes for the tests. You'd have that overhead in the very efficient c++ program too I suppose. ( https://github.com/nico/glitch/blob/master/glitch.go ) Nico
_______________________________________________ cfe-commits mailing list [email protected] http://lists.cs.uiuc.edu/mailman/listinfo/cfe-commits
