On Wed, Sep 18, 2019 at 10:46 AM Dawid Weiss <dawid.we...@gmail.com> wrote:
> > > If tests_jvms is a lucene/solr specific property, that needs to be in a > users "multi-project" "~/.gradle" file > > Yeah... I'm not at all a fan of modifying the global gradle settings > file. While it may be ok for settings that only apply to Lucene (if > prefixed) there are other things like enabling or disabling the daemon > which should be a matter of project-local settings. Ideally, I'd like > to see something like: I find the Gradle way of doing things odd, but I knew better than to try and go directly against them. So none of this is really my choosing, I went with how you are supposed to do it. When it comes to allowing another prop file just for our project, I'm all for it - just not the standard gradle.properties that is supposed to be committed in a gradle project. You cannot override any settings from your gradle home file regardless, so it's a little non intuitive - gradle home props rule all. Anyway, I have exposed like one of our properties, so no, it's not thought out or how it's going to be at all. Probably that property should be name spaced beyond tests_jvms (translated from tests.jvms), but then we might want to support a short cut notation for command line. Sucking in a user prop file like we do with ant sounds great, just have to get used to the fact that you cannot override your global settings. Trying to subvert that in some way def feels like a bad path. Anyway, I've never found it to be a problem. I didn't expect everyone would customize our project settings in their gradle home file, I usually put stuff there that I want to apply to all my checkouts (tests.jvms is actually one of them), but all that is very individual. You can also set these on the command line - it's mostly the same as ant. If you leave stuff out of your home gradle file and just add to some file we suck in, ezpeasy. I haven't prescribed or enforced anything here. There will be more/better getting started doc and more finished stuff the closer we get. > What's the plan as far as things like "ant beast", "-Dtests.dups", "-Dtests.iters" & "-Dtests.jvms" ? I've been having plenty of issues with just regular tests. Gradle statically assigns all tests in basically fs order to N JVMs. If you run tests against all the Solr modules, it's pretty sweet because it doesn't just use a few executors on the small modules and go a module at a time - it does them together more and less and keeps things busy. But running tests within a module, or even running all modules tests, was no good. You have solr-core tests coming late, you have many tests that pop up from short runs to minutes+ (often tests with many sub class tests), and you have a static JVM assignment for each test. So basically, it's easy to be running long tests at the end of a run and use few JVM's, and with our tests, this can be devastating. Now Dawids Ant test launcher tries to be a lot smarter about all this and caches run times so that it can order tests and holds some back for idle JVM's, etc. So I was looking at already very slow tests times on master often being much worse on gradle depending on if you ran tests across modules (which even the field a bit). We are a terrible case for gradles parallel tests. I figured I would improve tests times like usual to solve this as much as I could, but it's really not enough - our project is too large, too many tests, projects much smaller than our complain about this gradle issue. Luckily though, the abstractions and code that runs tests is super simple. So in the end, I made a new test target called testFast for now, and it's mostly feature complete with the test target (extends the Test task) except that it doesn't support rerunning just failed tests currently and doesn't support looking for tests via reflection, just file inclusions / exclusion regex. It's really just simple logic changes of a pretty simple process and gives us a lot of options. Of course we will leave gradles default test task in place and this will be an alternate target. The regular test task will find tests and send every single one in before any ever get processed, so that is changed to start processing right away instead (which is why the retry failed tests processor was taken out of the chain). So testFast scans all the tests and puts them into normal, slow, and slowest categories based on annotations. It shuffles those buckets and sends the slowest to the JVMs first. Then we hold tests back if too many are outstanding so that when we get towards the end, tests can go to idle JVM's instead of lining up early not knowing if it's a very slow line. When the tests hit the JVM's, instead of simple round robin, we look at how many tests are already in line at each JVM and choose the shortest line or shuffle. So I started with solr-core generally taking 20+ minutes or more on my 18 core machine, and with some other simple test taming, testFail can now do it in just under 7 minutes. Very easy to add the beasting support that is not already part of the test runner and easy to make sure the test runner can do it's job correctly. - Mark >
