> cool thanks for the pointer. I really like this list 😊 > The list is sort of internal detail... what I really wanted to have is a list of options and current "values" computed for a particular run of options and seed - this is the "testOpts" task that you can run for any project. Compare the output of these (note the flags - "C" for computed value, "!" for non-default value, etc.):
gradlew -p lucene/core testOpts gradlew -p lucene/solr testOpts gradlew -p lucene/core testOpts -Ptests.asserts=false This can be improved even more... and I have written a plugin that cleans up management of such options, but haven't had the time to port Lucene's build yet, eh. Dawid > So I am not sure at which point we ever had randomization of security > manager and/or asserts. I assume, Policeman Jenkins never had that. > > I have the feeling that Elastic did this on their build servers, but > that’s also not proved. I suggested that change in one of my talks at > BerlinBuzzwords, but may have never implemented it. > > > > Anyways: I am open to add randomization on Jenkins, it’s just 2 lines of > code in the randomize-java-groovy file on Policeman Jenkins. Maybe disable > asserts/and or SecurityManager in 1/5th of all cases. > > > > Uwe > > > > ----- > > Uwe Schindler > > Achterdiek 19, D-28357 Bremen > > https://www.thetaphi.de > > eMail: [email protected] > > > > *From:* Dawid Weiss <[email protected]> > *Sent:* Friday, February 19, 2021 7:52 PM > *To:* Lucene Dev <[email protected]> > *Subject:* Re: Random disabling of asserts in tests is not working > > > > > > Hi Uwe, > > > > No, it's not randomized - always runs with the security manager enabled. > All the options are here: > > > > > https://github.com/apache/lucene-solr/blob/master/gradle/testing/randomization.gradle#L68-L103 > > > > When the value says "random" we pick the random value at runtime (so that > it also works within IDEs). We could pick security manager at build-time > (derive from project seed). This is a no-brainer to do. As Robert said - > perhaps we should keep some things more strict for developers and just > shuffle on the CI-only. This requires passing -Ptests.*=... flags but is > simple, I think. > > > > Dawid > > > > On Fri, Feb 19, 2021 at 7:45 PM Uwe Schindler <[email protected]> wrote: > > Hi, > > I don’t fully remember what the setup previously was, but at least for > master and 8.x it does not automatically enable/disable asserts. We can of > course do this together with the other settings like GC or compressed OOPs, > its just a few more lines in the Groovy file. > > > > I was also thinking that we have Security Manager enabled/disabled from > time to time. But recently, I see no randomization for this on Jenkins, > unless it’s part of the Gradle build. > > > > Uwe > > > > ----- > > Uwe Schindler > > Achterdiek 19, D-28357 Bremen > > https://www.thetaphi.de > > eMail: [email protected] > > > > *From:* Robert Muir <[email protected]> > *Sent:* Friday, February 19, 2021 3:13 PM > *To:* [email protected] > *Subject:* Re: Random disabling of asserts in tests is not working > > > > I don't think it is enabled (at least in policeman jenkins). perhaps it > didn't work correctly when the build was cutover to gradle. Take a look at > any old build such as > https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/ > . You can see the variables it randomizes right there. > > > > You can confirm by clicking console->full log and it prints exact gradle > command that it runs: > https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/consoleFull > > > > Let's look into it, in a couple weeks or so? > > > > On Fri, Feb 19, 2021 at 8:32 AM Michael McCandless < > [email protected]> wrote: > > On Fri, Feb 19, 2021 at 8:07 AM Robert Muir <[email protected]> wrote: > > > > I think it has a downside: having a bug in an assert is really more of a > corner case. This is the kind of thing jenkins is for? > > > > Ahh, that is indeed a really good point. I would want/expect asserts to > always work correctly when running local tests ... if we randomly disabled > them in our checkouts it can cause a false sense of security, too soon. > > > > OK, I agree, let's leave it as randomization in Jenkins! How do we know > that Jenkins job/s are still randomizing assertions? Who tests the tester? > > > > Mike McCandless > > http://blog.mikemccandless.com > >
