I agree with Rob -- let's leave this sort of thing for Jenkins. It's really an edge case and I'd prefer consistently knowing the asserts always work locally.
~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Sat, Feb 20, 2021 at 1:14 PM Dawid Weiss <dawid.we...@gmail.com> wrote: > > cool thanks for the pointer. I really like this list 😊 >> > > The list is sort of internal detail... what I really wanted to have is a > list of options and current "values" computed for a particular run of > options and seed - this is the "testOpts" task that you can run for any > project. Compare the output of these (note the flags - "C" for computed > value, "!" for non-default value, etc.): > > gradlew -p lucene/core testOpts > gradlew -p lucene/solr testOpts > gradlew -p lucene/core testOpts -Ptests.asserts=false > > This can be improved even more... and I have written a plugin that cleans > up management of such options, but haven't had the time to port Lucene's > build yet, eh. > > Dawid > > > > >> So I am not sure at which point we ever had randomization of security >> manager and/or asserts. I assume, Policeman Jenkins never had that. >> >> I have the feeling that Elastic did this on their build servers, but >> that’s also not proved. I suggested that change in one of my talks at >> BerlinBuzzwords, but may have never implemented it. >> >> >> >> Anyways: I am open to add randomization on Jenkins, it’s just 2 lines of >> code in the randomize-java-groovy file on Policeman Jenkins. Maybe disable >> asserts/and or SecurityManager in 1/5th of all cases. >> >> >> >> Uwe >> >> >> >> ----- >> >> Uwe Schindler >> >> Achterdiek 19, D-28357 Bremen >> >> https://www.thetaphi.de >> >> eMail: u...@thetaphi.de >> >> >> >> *From:* Dawid Weiss <dawid.we...@gmail.com> >> *Sent:* Friday, February 19, 2021 7:52 PM >> *To:* Lucene Dev <dev@lucene.apache.org> >> *Subject:* Re: Random disabling of asserts in tests is not working >> >> >> >> >> >> Hi Uwe, >> >> >> >> No, it's not randomized - always runs with the security manager enabled. >> All the options are here: >> >> >> >> >> https://github.com/apache/lucene-solr/blob/master/gradle/testing/randomization.gradle#L68-L103 >> >> >> >> When the value says "random" we pick the random value at runtime (so that >> it also works within IDEs). We could pick security manager at build-time >> (derive from project seed). This is a no-brainer to do. As Robert said - >> perhaps we should keep some things more strict for developers and just >> shuffle on the CI-only. This requires passing -Ptests.*=... flags but is >> simple, I think. >> >> >> >> Dawid >> >> >> >> On Fri, Feb 19, 2021 at 7:45 PM Uwe Schindler <u...@thetaphi.de> wrote: >> >> Hi, >> >> I don’t fully remember what the setup previously was, but at least for >> master and 8.x it does not automatically enable/disable asserts. We can of >> course do this together with the other settings like GC or compressed OOPs, >> its just a few more lines in the Groovy file. >> >> >> >> I was also thinking that we have Security Manager enabled/disabled from >> time to time. But recently, I see no randomization for this on Jenkins, >> unless it’s part of the Gradle build. >> >> >> >> Uwe >> >> >> >> ----- >> >> Uwe Schindler >> >> Achterdiek 19, D-28357 Bremen >> >> https://www.thetaphi.de >> >> eMail: u...@thetaphi.de >> >> >> >> *From:* Robert Muir <rcm...@gmail.com> >> *Sent:* Friday, February 19, 2021 3:13 PM >> *To:* dev@lucene.apache.org >> *Subject:* Re: Random disabling of asserts in tests is not working >> >> >> >> I don't think it is enabled (at least in policeman jenkins). perhaps it >> didn't work correctly when the build was cutover to gradle. Take a look at >> any old build such as >> https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/ >> . You can see the variables it randomizes right there. >> >> >> >> You can confirm by clicking console->full log and it prints exact gradle >> command that it runs: >> https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/consoleFull >> >> >> >> Let's look into it, in a couple weeks or so? >> >> >> >> On Fri, Feb 19, 2021 at 8:32 AM Michael McCandless < >> luc...@mikemccandless.com> wrote: >> >> On Fri, Feb 19, 2021 at 8:07 AM Robert Muir <rcm...@gmail.com> wrote: >> >> >> >> I think it has a downside: having a bug in an assert is really more of a >> corner case. This is the kind of thing jenkins is for? >> >> >> >> Ahh, that is indeed a really good point. I would want/expect asserts to >> always work correctly when running local tests ... if we randomly disabled >> them in our checkouts it can cause a false sense of security, too soon. >> >> >> >> OK, I agree, let's leave it as randomization in Jenkins! How do we know >> that Jenkins job/s are still randomizing assertions? Who tests the tester? >> >> >> >> Mike McCandless >> >> http://blog.mikemccandless.com >> >>
