Re: Random disabling of asserts in tests is not working

[David Smiley]

I agree with Rob -- let's leave this sort of thing for Jenkins.  It's
really an edge case and I'd prefer consistently knowing the asserts always
work locally.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Sat, Feb 20, 2021 at 1:14 PM Dawid Weiss  wrote:

>
> cool thanks for the pointer. I really like this list 
>>
>
> The list is sort of internal detail... what I really wanted to have is a
> list of options and current "values" computed for a particular run of
> options and seed - this is the "testOpts" task that you can run for any
> project. Compare the output of these (note the flags - "C" for computed
> value, "!" for non-default value, etc.):
>
> gradlew -p lucene/core testOpts
> gradlew -p lucene/solr testOpts
> gradlew -p lucene/core testOpts -Ptests.asserts=false
>
> This can be improved even more... and I have written a plugin that cleans
> up management of such options, but haven't had the time to port Lucene's
> build yet, eh.
>
> Dawid
>
>
>
>
>> So I am not sure at which point we ever had randomization of security
>> manager and/or asserts. I assume, Policeman Jenkins never had that.
>>
>> I have the feeling that Elastic did this on their build servers, but
>> that’s also not proved. I suggested that change in one of my talks at
>> BerlinBuzzwords, but may have never implemented it.
>>
>>
>>
>> Anyways: I am open to add randomization on Jenkins, it’s just 2 lines of
>> code in the randomize-java-groovy file on Policeman Jenkins. Maybe disable
>> asserts/and or SecurityManager in 1/5th of all cases.
>>
>>
>>
>> Uwe
>>
>>
>>
>> -
>>
>> Uwe Schindler
>>
>> Achterdiek 19, D-28357 Bremen
>>
>> https://www.thetaphi.de
>>
>> eMail: u...@thetaphi.de
>>
>>
>>
>> *From:* Dawid Weiss 
>> *Sent:* Friday, February 19, 2021 7:52 PM
>> *To:* Lucene Dev 
>> *Subject:* Re: Random disabling of asserts in tests is not working
>>
>>
>>
>>
>>
>> Hi Uwe,
>>
>>
>>
>> No, it's not randomized - always runs with the security manager enabled.
>> All the options are here:
>>
>>
>>
>>
>> https://github.com/apache/lucene-solr/blob/master/gradle/testing/randomization.gradle#L68-L103
>>
>>
>>
>> When the value says "random" we pick the random value at runtime (so that
>> it also works within IDEs). We could pick security manager at build-time
>> (derive from project seed). This is a no-brainer to do. As Robert said -
>> perhaps we should keep some things more strict for developers and just
>> shuffle on the CI-only. This requires passing -Ptests.*=... flags but is
>> simple, I think.
>>
>>
>>
>> Dawid
>>
>>
>>
>> On Fri, Feb 19, 2021 at 7:45 PM Uwe Schindler  wrote:
>>
>> Hi,
>>
>> I don’t fully remember what the setup previously was, but at least for
>> master and 8.x it does not automatically enable/disable asserts. We can of
>> course do this together with the other settings like GC or compressed OOPs,
>> its just a few more lines in the Groovy file.
>>
>>
>>
>> I was also thinking that we have Security Manager enabled/disabled from
>> time to time. But recently, I see no randomization for this on Jenkins,
>> unless it’s part of the Gradle build.
>>
>>
>>
>> Uwe
>>
>>
>>
>> -
>>
>> Uwe Schindler
>>
>> Achterdiek 19, D-28357 Bremen
>>
>> https://www.thetaphi.de
>>
>> eMail: u...@thetaphi.de
>>
>>
>>
>> *From:* Robert Muir 
>> *Sent:* Friday, February 19, 2021 3:13 PM
>> *To:* dev@lucene.apache.org
>> *Subject:* Re: Random disabling of asserts in tests is not working
>>
>>
>>
>> I don't think it is enabled (at least in policeman jenkins). perhaps it
>> didn't work correctly when the build was cutover to gradle. Take a look at
>> any old build such as
>> https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/
>> . You can see the variables it randomizes right there.
>>
>>
>>
>> You can confirm by clicking console->full log and it prints exact gradle
>> command that it runs:
>> https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/consoleFull
>>
>>
>>
>> Let's look into it, in a couple weeks or so?
>>
>>
>>
>> On Fri, Feb 19, 2021 at 8:32 AM Michael McCandless <
>> luc...@mikemccandless.com> wrote:
>>
>> On Fri, Feb 19, 2021 at 8:07 AM Robert Muir  wrote:
>>
>>
>>
>> I think it has a downside: having a bug in an assert is really more of a
>> corner case. This is the kind of thing jenkins is for?
>>
>>
>>
>> Ahh, that is indeed a really good point.  I would want/expect asserts to
>> always work correctly when running local tests ... if we randomly disabled
>> them in our checkouts it can cause a false sense of security, too soon.
>>
>>
>>
>> OK, I agree, let's leave it as randomization in Jenkins!  How do we know
>> that Jenkins job/s are still randomizing assertions?  Who tests the tester?
>>
>>
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>

Re: OverseerStatusTest recent failures

[David Smiley]

Interesting.  Do you have a guess as to why the failures there are ~5% and
not 100% reproducible?

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Sat, Feb 20, 2021 at 6:41 PM Ilan Ginzburg  wrote:

> Indeed the issue is due to my changes.
>
> In OverseerStatusCmd I've skipped some stat collection when running in
> distributed cluster state updates mode because I thought these were only
> stats related to cluster state updates.
> Obviously that was too aggressive and some of the stats are related to the
> Collection API.
>
> I will make sure to skip returning only the stats that are related to
> cluster state updater and restore returning collection api stats (when
> running in distributed cluster updates mode, otherwise all stats are
> returned).
>
> Tomorrow...
>
> Ilan
>
> On Sun, Feb 21, 2021 at 12:22 AM Ilan Ginzburg  wrote:
>
>> Thank you David for reporting this.
>>
>> Seems due to my recent changes. I reproduce the failure locally and will
>> look at this tomorrow.
>>
>> With the distributed cluster state updates i've introduced a
>> randomization for using either Overseer based cluster state updates or
>> distributed cluster state updates in tests. This failure seems to happen in
>> the distributed state update case. I suspect it is due to Overseer
>> returning less stats than expected by the test (which is expected: Overseer
>> cannot return stats about cluster state updates if it does not handle
>> cluster state updates).
>>
>> The following line in the logs tells that the run is using distributed
>> cluster state:
>> 972874 INFO  (jetty-launcher-8973-thread-2) [ ]
>> o.a.s.c.DistributedClusterStateUpdater Creating
>> DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr
>> will be using distributed cluster state updates.
>>
>> Ilan
>>
>>
>> On Sat, Feb 20, 2021 at 3:00 PM David Smiley  wrote:
>>
>>> I encountered a failure from OverseerStatusTest locally.  According to
>>> our test failure trends, this guy only just recently started failing ~4-5%
>>> of the time, but previously was fine.  Only master branch.
>>>
>>>
>>> http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.OverseerStatusTest.test
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>

Re: Congratulations to the new Apache Solr PMC Chair, Jan Høydahl!

[Yonik Seeley]

Congrats Jan! Go Solr!
-Yonik


On Thu, Feb 18, 2021 at 1:56 PM Anshum Gupta  wrote:

> Hi everyone,
>
> I’d like to inform everyone that the newly formed Apache Solr PMC nominated
> and elected Jan Høydahl for the position of the Solr PMC Chair and Vice
> President. This decision was approved by the board in its February 2021
> meeting.
>
> Congratulations Jan!
>
> --
> Anshum Gupta
>

Re: Congratulations to the new Lucene PMC Chair, Michael Sokolov!

[Yonik Seeley]

Congrats Mike! Go Lucene!
-Yonik

On Wed, Feb 17, 2021 at 4:31 PM Anshum Gupta  wrote:

> Every year, the Lucene PMC rotates the Lucene PMC chair and Apache Vice
> President position.
>
> This year we nominated and elected Michael Sokolov as the Chair, a
> decision that the board approved in its February 2021 meeting.
>
> Congratulations, Mike!
>
> --
> Anshum Gupta
>

Re: Congratulations to the new Lucene PMC Chair, Michael Sokolov!

[Shalin Shekhar Mangar]

Congratulations, Mike and thank you!

On Thu, Feb 18, 2021 at 3:02 AM Anshum Gupta  wrote:

> Every year, the Lucene PMC rotates the Lucene PMC chair and Apache Vice
> President position.
>
> This year we nominated and elected Michael Sokolov as the Chair, a
> decision that the board approved in its February 2021 meeting.
>
> Congratulations, Mike!
>
> --
> Anshum Gupta
>


-- 
Regards,
Shalin Shekhar Mangar.

Re: Congratulations to the new Lucene PMC Chair, Michael Sokolov!

[Ilan Ginzburg]

Congratulations, Mike!

- Ilan

On Wed, Feb 17, 2021 at 10:32 PM Anshum Gupta 
wrote:

> Every year, the Lucene PMC rotates the Lucene PMC chair and Apache Vice
> President position.
>
> This year we nominated and elected Michael Sokolov as the Chair, a
> decision that the board approved in its February 2021 meeting.
>
> Congratulations, Mike!
>
> --
> Anshum Gupta
>

Re: OverseerStatusTest recent failures

[Ilan Ginzburg]

Indeed the issue is due to my changes.

In OverseerStatusCmd I've skipped some stat collection when running in
distributed cluster state updates mode because I thought these were only
stats related to cluster state updates.
Obviously that was too aggressive and some of the stats are related to the
Collection API.

I will make sure to skip returning only the stats that are related to
cluster state updater and restore returning collection api stats (when
running in distributed cluster updates mode, otherwise all stats are
returned).

Tomorrow...

Ilan

On Sun, Feb 21, 2021 at 12:22 AM Ilan Ginzburg  wrote:

> Thank you David for reporting this.
>
> Seems due to my recent changes. I reproduce the failure locally and will
> look at this tomorrow.
>
> With the distributed cluster state updates i've introduced a randomization
> for using either Overseer based cluster state updates or distributed
> cluster state updates in tests. This failure seems to happen in the
> distributed state update case. I suspect it is due to Overseer returning
> less stats than expected by the test (which is expected: Overseer cannot
> return stats about cluster state updates if it does not handle cluster
> state updates).
>
> The following line in the logs tells that the run is using distributed
> cluster state:
> 972874 INFO  (jetty-launcher-8973-thread-2) [ ]
> o.a.s.c.DistributedClusterStateUpdater Creating
> DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr
> will be using distributed cluster state updates.
>
> Ilan
>
>
> On Sat, Feb 20, 2021 at 3:00 PM David Smiley  wrote:
>
>> I encountered a failure from OverseerStatusTest locally.  According to
>> our test failure trends, this guy only just recently started failing ~4-5%
>> of the time, but previously was fine.  Only master branch.
>>
>>
>> http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.OverseerStatusTest.test
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>

Re: OverseerStatusTest recent failures

[Ilan Ginzburg]

Thank you David for reporting this.

Seems due to my recent changes. I reproduce the failure locally and will
look at this tomorrow.

With the distributed cluster state updates i've introduced a randomization
for using either Overseer based cluster state updates or distributed
cluster state updates in tests. This failure seems to happen in the
distributed state update case. I suspect it is due to Overseer returning
less stats than expected by the test (which is expected: Overseer cannot
return stats about cluster state updates if it does not handle cluster
state updates).

The following line in the logs tells that the run is using distributed
cluster state:
972874 INFO  (jetty-launcher-8973-thread-2) [ ]
o.a.s.c.DistributedClusterStateUpdater Creating
DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr
will be using distributed cluster state updates.

Ilan


On Sat, Feb 20, 2021 at 3:00 PM David Smiley  wrote:

> I encountered a failure from OverseerStatusTest locally.  According to our
> test failure trends, this guy only just recently started failing ~4-5% of
> the time, but previously was fine.  Only master branch.
>
>
> http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.OverseerStatusTest.test
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>

Re: Congratulations to the new Lucene PMC Chair, Michael Sokolov!

[Karl Wright]

Congratulations!


On Sat, Feb 20, 2021 at 4:17 PM Namgyu Kim  wrote:

> Congratulations, Mike! :D
>
> On Thu, Feb 18, 2021 at 6:32 AM Anshum Gupta 
> wrote:
>
>> Every year, the Lucene PMC rotates the Lucene PMC chair and Apache Vice
>> President position.
>>
>> This year we nominated and elected Michael Sokolov as the Chair, a
>> decision that the board approved in its February 2021 meeting.
>>
>> Congratulations, Mike!
>>
>> --
>> Anshum Gupta
>>
>

Re: Congratulations to the new Lucene PMC Chair, Michael Sokolov!

[Marcus Eagan]

Awesome. Congratulations Mike!

You truly have pushed open source search into a new era with Lucene-9004
and so many of your efforts to steward our community. You have inspired me
and others, I’m sure, to think about more innovative ways to contribute for
the future.

- Marcus

On Sat, Feb 20, 2021 at 16:18 Namgyu Kim  wrote:

> Congratulations, Mike! :D
>
> On Thu, Feb 18, 2021 at 6:32 AM Anshum Gupta 
> wrote:
>
>> Every year, the Lucene PMC rotates the Lucene PMC chair and Apache Vice
>> President position.
>>
>> This year we nominated and elected Michael Sokolov as the Chair, a
>> decision that the board approved in its February 2021 meeting.
>>
>> Congratulations, Mike!
>>
>> --
>> Anshum Gupta
>>
> --
Marcus Eagan

Simplifying source pattern checks

[Gus Heck]

I noticed today that SOLR-10883 added checks for patterns that didn't play
nice with PDF generation. Now that we don't generate the PDF anymore
perhaps we can do away with those checks? Anyone have thoughts to the
contrary?

https://issues.apache.org/jira/browse/SOLR-10883

-Gus

-- 
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)

Re: Congratulations to the new Apache Solr PMC Chair, Jan Høydahl!

[Namgyu Kim]

Congratulations, Jan! :D

On Fri, Feb 19, 2021 at 3:56 AM Anshum Gupta  wrote:

> Hi everyone,
>
> I’d like to inform everyone that the newly formed Apache Solr PMC
> nominated and elected Jan Høydahl for the position of the Solr PMC Chair
> and Vice President. This decision was approved by the board in its February
> 2021 meeting.
>
> Congratulations Jan!
>
> --
> Anshum Gupta
>

Re: Congratulations to the new Lucene PMC Chair, Michael Sokolov!

[Namgyu Kim]

Congratulations, Mike! :D

On Thu, Feb 18, 2021 at 6:32 AM Anshum Gupta  wrote:

> Every year, the Lucene PMC rotates the Lucene PMC chair and Apache Vice
> President position.
>
> This year we nominated and elected Michael Sokolov as the Chair, a
> decision that the board approved in its February 2021 meeting.
>
> Congratulations, Mike!
>
> --
> Anshum Gupta
>

Revisiting Standardized Test Names in Solr

[Marcus Eagan]

Hi all,

Now that Lucene’s standardization is complete and I believe enforced,
should we discuss if we could bring the same consistency to Solr?

Best,

Marcus
-- 
Marcus Eagan

Re: Congratulations to the new Apache Solr PMC Chair, Jan Høydahl!

[Noble Paul]

Congrats Jan

On Sun, Feb 21, 2021, 3:04 AM Marcus Eagan  wrote:

> Awesome, way to go Jan!
>
> - Marcus
>
> On Sat, Feb 20, 2021 at 10:53 Lucky Sharma  wrote:
>
>> Congratulations Jan
>>
>> Regards,
>> Lucky Sharma
>>
>> On Sat, 20 Feb 2021 at 8:07 PM, Karl Wright  wrote:
>>
>>> Congratulations!
>>> Karl
>>>
>>> On Sat, Feb 20, 2021 at 6:28 AM Uwe Schindler  wrote:
>>>
 Congrats Jan!



 Uwe



 -

 Uwe Schindler

 Achterdiek 19, D-28357 Bremen
 

 https://www.thetaphi.de

 eMail: u...@thetaphi.de



 *From:* Anshum Gupta 
 *Sent:* Thursday, February 18, 2021 7:55 PM
 *To:* Lucene Dev ; solr-u...@lucene.apache.org
 *Subject:* Congratulations to the new Apache Solr PMC Chair, Jan
 Høydahl!



 Hi everyone,



 I’d like to inform everyone that the newly formed Apache Solr PMC
 nominated and elected Jan Høydahl for the position of the Solr PMC Chair
 and Vice President. This decision was approved by the board in its February
 2021 meeting.



 Congratulations Jan!



 --

 Anshum Gupta

>>> --
>> Warm Regards,
>>
>> Lucky Sharma
>> Contact No :+91 9821559918
>>
> --
> Marcus Eagan
>
>

[RESULT] [VOTE] Release Lucene/Solr 8.8.1 RC2

[Timothy Potter]

It's been >72h since the vote was initiated and the result is:

+1  8  (7 binding)
 0  0
-1  0

This vote has PASSED


Thanks to those who tested the RC2!

Cheers,
Tim

PS ~ I have a rather busy weekend with non-work stuff, so I'm not sure
I'll be able to resume the release process until Monday AM US time.
Stay tuned ...

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Random disabling of asserts in tests is not working

[Dawid Weiss]

> cool thanks for the pointer. I really like this list 
>

The list is sort of internal detail... what I really wanted to have is a
list of options and current "values" computed for a particular run of
options and seed - this is the "testOpts" task that you can run for any
project. Compare the output of these (note the flags - "C" for computed
value, "!" for non-default value, etc.):

gradlew -p lucene/core testOpts
gradlew -p lucene/solr testOpts
gradlew -p lucene/core testOpts -Ptests.asserts=false

This can be improved even more... and I have written a plugin that cleans
up management of such options, but haven't had the time to port Lucene's
build yet, eh.

Dawid




> So I am not sure at which point we ever had randomization of security
> manager and/or asserts. I assume, Policeman Jenkins never had that.
>
> I have the feeling that Elastic did this on their build servers, but
> that’s also not proved. I suggested that change in one of my talks at
> BerlinBuzzwords, but may have never implemented it.
>
>
>
> Anyways: I am open to add randomization on Jenkins, it’s just 2 lines of
> code in the randomize-java-groovy file on Policeman Jenkins. Maybe disable
> asserts/and or SecurityManager in 1/5th of all cases.
>
>
>
> Uwe
>
>
>
> -
>
> Uwe Schindler
>
> Achterdiek 19, D-28357 Bremen
>
> https://www.thetaphi.de
>
> eMail: u...@thetaphi.de
>
>
>
> *From:* Dawid Weiss 
> *Sent:* Friday, February 19, 2021 7:52 PM
> *To:* Lucene Dev 
> *Subject:* Re: Random disabling of asserts in tests is not working
>
>
>
>
>
> Hi Uwe,
>
>
>
> No, it's not randomized - always runs with the security manager enabled.
> All the options are here:
>
>
>
>
> https://github.com/apache/lucene-solr/blob/master/gradle/testing/randomization.gradle#L68-L103
>
>
>
> When the value says "random" we pick the random value at runtime (so that
> it also works within IDEs). We could pick security manager at build-time
> (derive from project seed). This is a no-brainer to do. As Robert said -
> perhaps we should keep some things more strict for developers and just
> shuffle on the CI-only. This requires passing -Ptests.*=... flags but is
> simple, I think.
>
>
>
> Dawid
>
>
>
> On Fri, Feb 19, 2021 at 7:45 PM Uwe Schindler  wrote:
>
> Hi,
>
> I don’t fully remember what the setup previously was, but at least for
> master and 8.x it does not automatically enable/disable asserts. We can of
> course do this together with the other settings like GC or compressed OOPs,
> its just a few more lines in the Groovy file.
>
>
>
> I was also thinking that we have Security Manager enabled/disabled from
> time to time. But recently, I see no randomization for this on Jenkins,
> unless it’s part of the Gradle build.
>
>
>
> Uwe
>
>
>
> -
>
> Uwe Schindler
>
> Achterdiek 19, D-28357 Bremen
>
> https://www.thetaphi.de
>
> eMail: u...@thetaphi.de
>
>
>
> *From:* Robert Muir 
> *Sent:* Friday, February 19, 2021 3:13 PM
> *To:* dev@lucene.apache.org
> *Subject:* Re: Random disabling of asserts in tests is not working
>
>
>
> I don't think it is enabled (at least in policeman jenkins). perhaps it
> didn't work correctly when the build was cutover to gradle. Take a look at
> any old build such as
> https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/
> . You can see the variables it randomizes right there.
>
>
>
> You can confirm by clicking console->full log and it prints exact gradle
> command that it runs:
> https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/consoleFull
>
>
>
> Let's look into it, in a couple weeks or so?
>
>
>
> On Fri, Feb 19, 2021 at 8:32 AM Michael McCandless <
> luc...@mikemccandless.com> wrote:
>
> On Fri, Feb 19, 2021 at 8:07 AM Robert Muir  wrote:
>
>
>
> I think it has a downside: having a bug in an assert is really more of a
> corner case. This is the kind of thing jenkins is for?
>
>
>
> Ahh, that is indeed a really good point.  I would want/expect asserts to
> always work correctly when running local tests ... if we randomly disabled
> them in our checkouts it can cause a false sense of security, too soon.
>
>
>
> OK, I agree, let's leave it as randomization in Jenkins!  How do we know
> that Jenkins job/s are still randomizing assertions?  Who tests the tester?
>
>
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>

Re: Congratulations to the new Apache Solr PMC Chair, Jan Høydahl!

[Marcus Eagan]

Awesome, way to go Jan!

- Marcus

On Sat, Feb 20, 2021 at 10:53 Lucky Sharma  wrote:

> Congratulations Jan
>
> Regards,
> Lucky Sharma
>
> On Sat, 20 Feb 2021 at 8:07 PM, Karl Wright  wrote:
>
>> Congratulations!
>> Karl
>>
>> On Sat, Feb 20, 2021 at 6:28 AM Uwe Schindler  wrote:
>>
>>> Congrats Jan!
>>>
>>>
>>>
>>> Uwe
>>>
>>>
>>>
>>> -
>>>
>>> Uwe Schindler
>>>
>>> Achterdiek 19, D-28357 Bremen
>>> 
>>>
>>> https://www.thetaphi.de
>>>
>>> eMail: u...@thetaphi.de
>>>
>>>
>>>
>>> *From:* Anshum Gupta 
>>> *Sent:* Thursday, February 18, 2021 7:55 PM
>>> *To:* Lucene Dev ; solr-u...@lucene.apache.org
>>> *Subject:* Congratulations to the new Apache Solr PMC Chair, Jan
>>> Høydahl!
>>>
>>>
>>>
>>> Hi everyone,
>>>
>>>
>>>
>>> I’d like to inform everyone that the newly formed Apache Solr PMC
>>> nominated and elected Jan Høydahl for the position of the Solr PMC Chair
>>> and Vice President. This decision was approved by the board in its February
>>> 2021 meeting.
>>>
>>>
>>>
>>> Congratulations Jan!
>>>
>>>
>>>
>>> --
>>>
>>> Anshum Gupta
>>>
>> --
> Warm Regards,
>
> Lucky Sharma
> Contact No :+91 9821559918
>
-- 
Marcus Eagan

Re: Congratulations to the new Apache Solr PMC Chair, Jan Høydahl!

[Lucky Sharma]

Congratulations Jan

Regards,
Lucky Sharma

On Sat, 20 Feb 2021 at 8:07 PM, Karl Wright  wrote:

> Congratulations!
> Karl
>
> On Sat, Feb 20, 2021 at 6:28 AM Uwe Schindler  wrote:
>
>> Congrats Jan!
>>
>>
>>
>> Uwe
>>
>>
>>
>> -
>>
>> Uwe Schindler
>>
>> Achterdiek 19, D-28357 Bremen
>> 
>>
>> https://www.thetaphi.de
>>
>> eMail: u...@thetaphi.de
>>
>>
>>
>> *From:* Anshum Gupta 
>> *Sent:* Thursday, February 18, 2021 7:55 PM
>> *To:* Lucene Dev ; solr-u...@lucene.apache.org
>> *Subject:* Congratulations to the new Apache Solr PMC Chair, Jan Høydahl!
>>
>>
>>
>> Hi everyone,
>>
>>
>>
>> I’d like to inform everyone that the newly formed Apache Solr PMC
>> nominated and elected Jan Høydahl for the position of the Solr PMC Chair
>> and Vice President. This decision was approved by the board in its February
>> 2021 meeting.
>>
>>
>>
>> Congratulations Jan!
>>
>>
>>
>> --
>>
>> Anshum Gupta
>>
> --
Warm Regards,

Lucky Sharma
Contact No :+91 9821559918

Re: Congratulations to the new Apache Solr PMC Chair, Jan Høydahl!

[Karl Wright]

Congratulations!
Karl

On Sat, Feb 20, 2021 at 6:28 AM Uwe Schindler  wrote:

> Congrats Jan!
>
>
>
> Uwe
>
>
>
> -
>
> Uwe Schindler
>
> Achterdiek 19, D-28357 Bremen
>
> https://www.thetaphi.de
>
> eMail: u...@thetaphi.de
>
>
>
> *From:* Anshum Gupta 
> *Sent:* Thursday, February 18, 2021 7:55 PM
> *To:* Lucene Dev ; solr-u...@lucene.apache.org
> *Subject:* Congratulations to the new Apache Solr PMC Chair, Jan Høydahl!
>
>
>
> Hi everyone,
>
>
>
> I’d like to inform everyone that the newly formed Apache Solr PMC
> nominated and elected Jan Høydahl for the position of the Solr PMC Chair
> and Vice President. This decision was approved by the board in its February
> 2021 meeting.
>
>
>
> Congratulations Jan!
>
>
>
> --
>
> Anshum Gupta
>

OverseerStatusTest recent failures

[David Smiley]

I encountered a failure from OverseerStatusTest locally.  According to our
test failure trends, this guy only just recently started failing ~4-5% of
the time, but previously was fine.  Only master branch.

http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.OverseerStatusTest.test

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley

RE: Congratulations to the new Apache Solr PMC Chair, Jan Høydahl!

[Uwe Schindler]

Congrats Jan!

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

  https://www.thetaphi.de

eMail: u...@thetaphi.de

 

From: Anshum Gupta  
Sent: Thursday, February 18, 2021 7:55 PM
To: Lucene Dev ; solr-u...@lucene.apache.org
Subject: Congratulations to the new Apache Solr PMC Chair, Jan Høydahl!

 

Hi everyone,

 

I’d like to inform everyone that the newly formed Apache Solr PMC nominated and 
elected Jan Høydahl for the position of the Solr PMC Chair and Vice President. 
This decision was approved by the board in its February 2021 meeting.

 

Congratulations Jan! 

 

-- 

Anshum Gupta

RE: Random disabling of asserts in tests is not working

[Uwe Schindler]

Hi Dawid,

 

cool thanks for the pointer. I really like this list 

So I am not sure at which point we ever had randomization of security manager 
and/or asserts. I assume, Policeman Jenkins never had that.

I have the feeling that Elastic did this on their build servers, but that’s 
also not proved. I suggested that change in one of my talks at BerlinBuzzwords, 
but may have never implemented it.

 

Anyways: I am open to add randomization on Jenkins, it’s just 2 lines of code 
in the randomize-java-groovy file on Policeman Jenkins. Maybe disable 
asserts/and or SecurityManager in 1/5th of all cases. 

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

https://www.thetaphi.de

eMail: u...@thetaphi.de

 

From: Dawid Weiss  
Sent: Friday, February 19, 2021 7:52 PM
To: Lucene Dev 
Subject: Re: Random disabling of asserts in tests is not working

 

 

Hi Uwe,

 

No, it's not randomized - always runs with the security manager enabled. All 
the options are here:

 

https://github.com/apache/lucene-solr/blob/master/gradle/testing/randomization.gradle#L68-L103

 

When the value says "random" we pick the random value at runtime (so that it 
also works within IDEs). We could pick security manager at build-time (derive 
from project seed). This is a no-brainer to do. As Robert said - perhaps we 
should keep some things more strict for developers and just shuffle on the 
CI-only. This requires passing -Ptests.*=... flags but is simple, I think.

 

Dawid

 

On Fri, Feb 19, 2021 at 7:45 PM Uwe Schindler mailto:u...@thetaphi.de> > wrote:

Hi,

I don’t fully remember what the setup previously was, but at least for master 
and 8.x it does not automatically enable/disable asserts. We can of course do 
this together with the other settings like GC or compressed OOPs, its just a 
few more lines in the Groovy file.

 

I was also thinking that we have Security Manager enabled/disabled from time to 
time. But recently, I see no randomization for this on Jenkins, unless it’s 
part of the Gradle build.

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

https://www.thetaphi.de

eMail: u...@thetaphi.de  

 

From: Robert Muir mailto:rcm...@gmail.com> > 
Sent: Friday, February 19, 2021 3:13 PM
To: dev@lucene.apache.org  
Subject: Re: Random disabling of asserts in tests is not working

 

I don't think it is enabled (at least in policeman jenkins). perhaps it didn't 
work correctly when the build was cutover to gradle. Take a look at any old 
build such as 
https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/
 . You can see the variables it randomizes right there.

 

You can confirm by clicking console->full log and it prints exact gradle 
command that it runs: 
https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/29491/consoleFull

 

Let's look into it, in a couple weeks or so? 

 

On Fri, Feb 19, 2021 at 8:32 AM Michael McCandless mailto:luc...@mikemccandless.com> > wrote:

On Fri, Feb 19, 2021 at 8:07 AM Robert Muir mailto:rcm...@gmail.com> > wrote:

 

I think it has a downside: having a bug in an assert is really more of a corner 
case. This is the kind of thing jenkins is for?

 

Ahh, that is indeed a really good point.  I would want/expect asserts to always 
work correctly when running local tests ... if we randomly disabled them in our 
checkouts it can cause a false sense of security, too soon.

 

OK, I agree, let's leave it as randomization in Jenkins!  How do we know that 
Jenkins job/s are still randomizing assertions?  Who tests the tester?

 

Mike McCandless

http://blog.mikemccandless.com

Re: GC cost of creating String resource description on IndexInput clone

[Robert Muir]

+1 to explore Uwe's lambda idea. I really hate lambdas (you should too, if
you know what happens behind the scenes), but this sounds like maybe an
actual justified use of lambdas, taking advantage of the capture facility
and not using them just to be "cute"

On Sat, Feb 20, 2021 at 6:17 AM Uwe Schindler  wrote:

> Hi,
>
>
>
> I was thinking about something similar when working on the Java 17
> replacement for MMapDirectory. My idea would still be late evaluation, but
> using a lambda. Same way how you can do lazy evaluation in Log4j2 when
> logging with debug/trace/…
>
>
>
> Instead of having “String resourceDescription“ as a field in each
> Indexinput, we can use “Supplier resourceDescription” and
> initialize it like: () -> foo + bar + parent.toString +…
>
>
>
> Just because this might be an option to do, I fully agree with Mike Drob
> and Robert Muir: The risk of memory leaks for such a small improvement is
> too high. It also depends on the GC you use. If you use old
> ConcMarkSweepGC, I agree this is an overhead, but nowadays, Lucene should
> use G1GC.
>
>
>
> In addition, the strings are not for debugging, they are really useful
> when something goes wrong, e.g. when I/O errors occur.
>
>
>
> Uwe
>
>
>
> -
>
> Uwe Schindler
>
> Achterdiek 19, D-28357 Bremen
>
> https://www.thetaphi.de
>
> eMail: u...@thetaphi.de
>
>
>
> *From:* Robert Muir 
> *Sent:* Saturday, February 20, 2021 11:56 AM
> *To:* dev@lucene.apache.org
> *Subject:* Re: GC cost of creating String resource description on
> IndexInput clone
>
>
>
> The issue is that clone or not, they are both IndexInput.java. So if we go
> with your proposal, then *sometimes* the code will have this reference and
> *other times* it won't and the reference will be null. In that non-clone
> case, where would its resource description (filename) come from?  I predict
> too many bugs.
>
>
>
> Making this part of the code complex would be the wrong tradeoff for 1%
> performance.
>
>
>
>
>
> On Fri, Feb 19, 2021 at 4:58 PM Viral Gandhi 
> wrote:
>
> Hello everyone,
>
> We recently added Java Flight Recorder (JFR) to our internal benchmarking.
> While looking through top heap allocations from JFR output, we noticed the
> following to be in top 5 contributors to garbage creation.
>
> org.apache.lucene.store.IndexInput#getFullSliceDescription()
>   at org.apache.lucene.store.ByteBufferIndexInput#buildSlice()
>at org.apache.lucene.store.ByteBufferIndexInput#slice()
>at
> org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl#slice()
>at org.apache.lucene.store.IndexInput#randomAccessSlice()
>
> It seems that we construct a new String resource description for each
> clone/slice of IndexInput instance. Resource description for a clone/slice
> instance is a String concatenation of resource description from original
> instance and current slice description. Also, clone/slice can happen
> multiple times per query per segment.
>
> Can we avoid upfront String allocation by late-evaluating
> IndexInput.toString() on the cloned instance? One approach could be to
> hold a reference of the base IndexInput instance in the cloned instance.
> Then while evaluating IndexInput.toString() on a cloned instance, we can
> construct a resource description by concatenating base instance's
> toString() with current resource description. My understanding is that
> IndexInput.toString() is primarily used for debugging purposes hence we
> can benefit from the late-evaluation.
>
> With this approach, we are seeing sustainable GC time reduction of ~6% and
> gain of ~1% to red-line QPS (throughput). My intention to start this thread
> is to collect feedback on this approach as well as to discuss any other
> ideas.
>
>
>
> Thanks,
>
> Viral Gandhi
>
>

RE: GC cost of creating String resource description on IndexInput clone

[Uwe Schindler]

Hi,

 

On top of that: You did not prove that the resourceDescription String in 
IndexInput is the main GC problem. I would suspect that actually the duplicated 
ByteBuffers are the main contributor to GC; those are unavoidable! The String 
is produced together with the ByteBuffer duplicates, so I would say: you can’t 
figure out from JFR what’s the main contributor (the small strings in between 
should be optimized away anyways).

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

https://www.thetaphi.de

eMail: u...@thetaphi.de

 

From: Uwe Schindler  
Sent: Saturday, February 20, 2021 12:17 PM
To: 'dev@lucene.apache.org' 
Subject: RE: GC cost of creating String resource description on IndexInput clone

 

Hi,

 

I was thinking about something similar when working on the Java 17 replacement 
for MMapDirectory. My idea would still be late evaluation, but using a lambda. 
Same way how you can do lazy evaluation in Log4j2 when logging with 
debug/trace/…

 

Instead of having “String resourceDescription“ as a field in each Indexinput, 
we can use “Supplier resourceDescription” and initialize it like: () -> 
foo + bar + parent.toString +…

 

Just because this might be an option to do, I fully agree with Mike Drob and 
Robert Muir: The risk of memory leaks for such a small improvement is too high. 
It also depends on the GC you use. If you use old ConcMarkSweepGC, I agree this 
is an overhead, but nowadays, Lucene should use G1GC.

 

In addition, the strings are not for debugging, they are really useful when 
something goes wrong, e.g. when I/O errors occur.

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

https://www.thetaphi.de

eMail: u...@thetaphi.de  

 

From: Robert Muir mailto:rcm...@gmail.com> > 
Sent: Saturday, February 20, 2021 11:56 AM
To: dev@lucene.apache.org  
Subject: Re: GC cost of creating String resource description on IndexInput clone

 

The issue is that clone or not, they are both IndexInput.java. So if we go with 
your proposal, then *sometimes* the code will have this reference and *other 
times* it won't and the reference will be null. In that non-clone case, where 
would its resource description (filename) come from?  I predict too many bugs. 

 

Making this part of the code complex would be the wrong tradeoff for 1% 
performance.

 

 

On Fri, Feb 19, 2021 at 4:58 PM Viral Gandhi mailto:viral.dev...@gmail.com> > wrote:

Hello everyone,

We recently added Java Flight Recorder (JFR) to our internal benchmarking. 
While looking through top heap allocations from JFR output, we noticed the 
following to be in top 5 contributors to garbage creation.

org.apache.lucene.store.IndexInput#getFullSliceDescription()
  at org.apache.lucene.store.ByteBufferIndexInput#buildSlice()
   at org.apache.lucene.store.ByteBufferIndexInput#slice()
   at org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl#slice()
   at org.apache.lucene.store.IndexInput#randomAccessSlice()

It seems that we construct a new String resource description for each 
clone/slice of IndexInput instance. Resource description for a clone/slice 
instance is a String concatenation of resource description from original 
instance and current slice description. Also, clone/slice can happen multiple 
times per query per segment. 

Can we avoid upfront String allocation by late-evaluating IndexInput.toString() 
on the cloned instance? One approach could be to hold a reference of the base 
IndexInput instance in the cloned instance. Then while evaluating 
IndexInput.toString() on a cloned instance, we can construct a resource 
description by concatenating base instance's toString() with current resource 
description. My understanding is that IndexInput.toString() is primarily used 
for debugging purposes hence we can benefit from the late-evaluation. 

With this approach, we are seeing sustainable GC time reduction of ~6% and gain 
of ~1% to red-line QPS (throughput). My intention to start this thread is to 
collect feedback on this approach as well as to discuss any other ideas.

 

Thanks,

Viral Gandhi

RE: GC cost of creating String resource description on IndexInput clone

[Uwe Schindler]

Hi,

 

I was thinking about something similar when working on the Java 17 replacement 
for MMapDirectory. My idea would still be late evaluation, but using a lambda. 
Same way how you can do lazy evaluation in Log4j2 when logging with 
debug/trace/…

 

Instead of having “String resourceDescription“ as a field in each Indexinput, 
we can use “Supplier resourceDescription” and initialize it like: () -> 
foo + bar + parent.toString +…

 

Just because this might be an option to do, I fully agree with Mike Drob and 
Robert Muir: The risk of memory leaks for such a small improvement is too high. 
It also depends on the GC you use. If you use old ConcMarkSweepGC, I agree this 
is an overhead, but nowadays, Lucene should use G1GC.

 

In addition, the strings are not for debugging, they are really useful when 
something goes wrong, e.g. when I/O errors occur.

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

https://www.thetaphi.de

eMail: u...@thetaphi.de

 

From: Robert Muir  
Sent: Saturday, February 20, 2021 11:56 AM
To: dev@lucene.apache.org
Subject: Re: GC cost of creating String resource description on IndexInput clone

 

The issue is that clone or not, they are both IndexInput.java. So if we go with 
your proposal, then *sometimes* the code will have this reference and *other 
times* it won't and the reference will be null. In that non-clone case, where 
would its resource description (filename) come from?  I predict too many bugs. 

 

Making this part of the code complex would be the wrong tradeoff for 1% 
performance.

 

 

On Fri, Feb 19, 2021 at 4:58 PM Viral Gandhi mailto:viral.dev...@gmail.com> > wrote:

Hello everyone,

We recently added Java Flight Recorder (JFR) to our internal benchmarking. 
While looking through top heap allocations from JFR output, we noticed the 
following to be in top 5 contributors to garbage creation.

org.apache.lucene.store.IndexInput#getFullSliceDescription()
  at org.apache.lucene.store.ByteBufferIndexInput#buildSlice()
   at org.apache.lucene.store.ByteBufferIndexInput#slice()
   at org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl#slice()
   at org.apache.lucene.store.IndexInput#randomAccessSlice()

It seems that we construct a new String resource description for each 
clone/slice of IndexInput instance. Resource description for a clone/slice 
instance is a String concatenation of resource description from original 
instance and current slice description. Also, clone/slice can happen multiple 
times per query per segment. 

Can we avoid upfront String allocation by late-evaluating IndexInput.toString() 
on the cloned instance? One approach could be to hold a reference of the base 
IndexInput instance in the cloned instance. Then while evaluating 
IndexInput.toString() on a cloned instance, we can construct a resource 
description by concatenating base instance's toString() with current resource 
description. My understanding is that IndexInput.toString() is primarily used 
for debugging purposes hence we can benefit from the late-evaluation. 

With this approach, we are seeing sustainable GC time reduction of ~6% and gain 
of ~1% to red-line QPS (throughput). My intention to start this thread is to 
collect feedback on this approach as well as to discuss any other ideas.

 

Thanks,

Viral Gandhi

Re: GC cost of creating String resource description on IndexInput clone

[Robert Muir]

The issue is that clone or not, they are both IndexInput.java. So if we go
with your proposal, then *sometimes* the code will have this reference and
*other times* it won't and the reference will be null. In that non-clone
case, where would its resource description (filename) come from?  I predict
too many bugs.

Making this part of the code complex would be the wrong tradeoff for 1%
performance.


On Fri, Feb 19, 2021 at 4:58 PM Viral Gandhi  wrote:

> Hello everyone,
>
> We recently added Java Flight Recorder (JFR) to our internal benchmarking.
> While looking through top heap allocations from JFR output, we noticed the
> following to be in top 5 contributors to garbage creation.
>
> org.apache.lucene.store.IndexInput#getFullSliceDescription()
>   at org.apache.lucene.store.ByteBufferIndexInput#buildSlice()
>at org.apache.lucene.store.ByteBufferIndexInput#slice()
>at
> org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl#slice()
>at org.apache.lucene.store.IndexInput#randomAccessSlice()
>
> It seems that we construct a new String resource description for each
> clone/slice of IndexInput instance. Resource description for a clone/slice
> instance is a String concatenation of resource description from original
> instance and current slice description. Also, clone/slice can happen
> multiple times per query per segment.
>
> Can we avoid upfront String allocation by late-evaluating
> IndexInput.toString() on the cloned instance? One approach could be to
> hold a reference of the base IndexInput instance in the cloned instance.
> Then while evaluating IndexInput.toString() on a cloned instance, we can
> construct a resource description by concatenating base instance's
> toString() with current resource description. My understanding is that
> IndexInput.toString() is primarily used for debugging purposes hence we
> can benefit from the late-evaluation.
>
> With this approach, we are seeing sustainable GC time reduction of ~6% and
> gain of ~1% to red-line QPS (throughput). My intention to start this thread
> is to collect feedback on this approach as well as to discuss any other
> ideas.
>
> Thanks,
> Viral Gandhi
>