Re: SolrCloud is sick.

[Mark Miller]

Sorry you guys got to play therapist for a bit. 10 years to get over, most
of it buried. Had to let that beast out.

On Wed, Nov 6, 2019 at 12:33 AM Mark Miller  wrote:

> Another Overseer :)
>
> I don't mean to contradict your curator statement either - I talk with the
> authority of god but with the confidence of no one ;)
>
> On Tue, Nov 5, 2019 at 7:44 PM Scott Blum  wrote:
>
>> WHO OVERSEES THE OVERSEER
>>
>> On Tue, Nov 5, 2019 at 5:16 PM Mark Miller  wrote:
>>
>>> bq.   SolrCloud is a ballerina. Doesn't look it, cause we dont take care
>>> of it.
>>>
>>> And this is why I'm so devastated by the Overseer. I don't blame anyone
>>> person. Where was the manual, where was my intervention. I whispered
>>> Overseer and cut one more thing off my list of responsibilities.
>>>
>>> But the overseer is supposed to be so light weight and easy breezy.
>>> Giving up leader shop at the most signs of trouble, keeping
>>> communication small and tight with tiny json distrib queue pub/sub updates.
>>> Little about stat change, hardly needed. Hardly ever talking to Zookeeper.
>>>
>>> Our whole system is not moved hard against this, but nothing so much as
>>> the Overseer. It has very scary, very tricky, custom ZK code. It has major
>>> communication with ZK. It has little to weak ability to properly throttle
>>> itself or deal with things intelligently. It's almost a brute force tactic.
>>> And it clings to being Overseer like a moth to flame. It's designed to be
>>> on a dedicated hardwar and then mostly to not make any reasonable use of
>>> that hardware.
>>>
>>> I blame me more than anyone for that. I am mad at me. It's just an
>>> absolute brain bash with a sledge hammer to the system. And i never
>>> communicated the system very well. I was overloaded.
>>>
>>> On Tue, Nov 5, 2019 at 11:01 AM Mark Miller 
>>> wrote:
>>>
 And now we are to meat of it. Fill in here:
 https://issues.apache.org/jira/browse/SOLR-13888

 We can play in a better world, we can have fun, but some of you are
 going to have to adjust your ways. In the most convenient way possible. You
 are all great people, I don't want to cause you annoyance, but there are
 certain requirements to building an aircraft, and there certain
 requirements to building this.

 On Tue, Nov 5, 2019 at 10:44 AM Mark Miller 
 wrote:

> If you had any idea how much suffering just that has caused. Not just
> users, but us.
>
> Mark
>
> On Tue, Nov 5, 2019 at 10:38 AM Mark Miller 
> wrote:
>
>> It’s like 6-7 years since I quickly added a shitty collections API in
>> my free time because we desperately needed SOMETHING. I don’t know if I
>> tried to make it wait for proper state or what , it was a stub to try get
>> things moving. That call, to this day, along with all our other checks,
>> until some tests ones recently, is garbage.
>>
>> If I downloaded a database, and a lot the time, after the create a
>> database call returned, my database was not ready, I’d saw wow. Terrible
>> bug got through. If it was a persistent issue for over half a decade? My
>> god.
>>
>> Look I just spent that half decade upgrading from Solr 4 to whatever.
>> I was mostly out of the loop. But this is crazy, me in there too.
>>
>> Mark
>>
>> On Tue, Nov 5, 2019 at 10:05 AM Mark Miller 
>> wrote:
>>
>>> I'll tell you what guys, development right now sucks. I don't enjoy.
>>>
>>> But when I start to put things in shape? I get this smile, and I
>>> start going with the feeling of I don't need you guys, I don't users, I
>>> don't need a job, cause just this is figgen nice.
>>>
>>> On Tue, Nov 5, 2019 at 9:59 AM Mark Miller 
>>> wrote:
>>>
 I suppose I should toss one more out.

 Hell yes, we will be using curator.

 It's insane for any group larger than 2-3 to directly use
 ZooKeeper. Even for that group, you want some damn good reasons to not 
 use
 curator. We can start using more assembly too (joke Yonik).

 Curator was an option initially. Then it was yet another project
 hosted by Netflix. Now it is essential.


 - Mark

 On Tue, Nov 5, 2019 at 9:41 AM Mark Miller 
 wrote:

> And look, we started pretty deep in the hole. Solr started with
> tons of bug or limitations that hardly mattered to it and hit 
> SolrCloud in
> the eye like a train. And we were not setup to deal with that.
>
> We never had a nice garden for SolrCloud. We started in a mess,
> thinking, eventually we clear the overgrowth, and we are all good. 
> And then
> we started building our house and that garden went wild with a life 
> of it's
> own.
>
> And our development practices, 

Re: SolrCloud is sick.

[Mark Miller]

Another Overseer :)

I don't mean to contradict your curator statement either - I talk with the
authority of god but with the confidence of no one ;)

On Tue, Nov 5, 2019 at 7:44 PM Scott Blum  wrote:

> WHO OVERSEES THE OVERSEER
>
> On Tue, Nov 5, 2019 at 5:16 PM Mark Miller  wrote:
>
>> bq.   SolrCloud is a ballerina. Doesn't look it, cause we dont take care
>> of it.
>>
>> And this is why I'm so devastated by the Overseer. I don't blame anyone
>> person. Where was the manual, where was my intervention. I whispered
>> Overseer and cut one more thing off my list of responsibilities.
>>
>> But the overseer is supposed to be so light weight and easy breezy.
>> Giving up leader shop at the most signs of trouble, keeping
>> communication small and tight with tiny json distrib queue pub/sub updates.
>> Little about stat change, hardly needed. Hardly ever talking to Zookeeper.
>>
>> Our whole system is not moved hard against this, but nothing so much as
>> the Overseer. It has very scary, very tricky, custom ZK code. It has major
>> communication with ZK. It has little to weak ability to properly throttle
>> itself or deal with things intelligently. It's almost a brute force tactic.
>> And it clings to being Overseer like a moth to flame. It's designed to be
>> on a dedicated hardwar and then mostly to not make any reasonable use of
>> that hardware.
>>
>> I blame me more than anyone for that. I am mad at me. It's just an
>> absolute brain bash with a sledge hammer to the system. And i never
>> communicated the system very well. I was overloaded.
>>
>> On Tue, Nov 5, 2019 at 11:01 AM Mark Miller 
>> wrote:
>>
>>> And now we are to meat of it. Fill in here:
>>> https://issues.apache.org/jira/browse/SOLR-13888
>>>
>>> We can play in a better world, we can have fun, but some of you are
>>> going to have to adjust your ways. In the most convenient way possible. You
>>> are all great people, I don't want to cause you annoyance, but there are
>>> certain requirements to building an aircraft, and there certain
>>> requirements to building this.
>>>
>>> On Tue, Nov 5, 2019 at 10:44 AM Mark Miller 
>>> wrote:
>>>
 If you had any idea how much suffering just that has caused. Not just
 users, but us.

 Mark

 On Tue, Nov 5, 2019 at 10:38 AM Mark Miller 
 wrote:

> It’s like 6-7 years since I quickly added a shitty collections API in
> my free time because we desperately needed SOMETHING. I don’t know if I
> tried to make it wait for proper state or what , it was a stub to try get
> things moving. That call, to this day, along with all our other checks,
> until some tests ones recently, is garbage.
>
> If I downloaded a database, and a lot the time, after the create a
> database call returned, my database was not ready, I’d saw wow. Terrible
> bug got through. If it was a persistent issue for over half a decade? My
> god.
>
> Look I just spent that half decade upgrading from Solr 4 to whatever.
> I was mostly out of the loop. But this is crazy, me in there too.
>
> Mark
>
> On Tue, Nov 5, 2019 at 10:05 AM Mark Miller 
> wrote:
>
>> I'll tell you what guys, development right now sucks. I don't enjoy.
>>
>> But when I start to put things in shape? I get this smile, and I
>> start going with the feeling of I don't need you guys, I don't users, I
>> don't need a job, cause just this is figgen nice.
>>
>> On Tue, Nov 5, 2019 at 9:59 AM Mark Miller 
>> wrote:
>>
>>> I suppose I should toss one more out.
>>>
>>> Hell yes, we will be using curator.
>>>
>>> It's insane for any group larger than 2-3 to directly use ZooKeeper.
>>> Even for that group, you want some damn good reasons to not use 
>>> curator. We
>>> can start using more assembly too (joke Yonik).
>>>
>>> Curator was an option initially. Then it was yet another project
>>> hosted by Netflix. Now it is essential.
>>>
>>>
>>> - Mark
>>>
>>> On Tue, Nov 5, 2019 at 9:41 AM Mark Miller 
>>> wrote:
>>>
 And look, we started pretty deep in the hole. Solr started with
 tons of bug or limitations that hardly mattered to it and hit 
 SolrCloud in
 the eye like a train. And we were not setup to deal with that.

 We never had a nice garden for SolrCloud. We started in a mess,
 thinking, eventually we clear the overgrowth, and we are all good. And 
 then
 we started building our house and that garden went wild with a life of 
 it's
 own.

 And our development practices, amazingly above many many many
 groups and standards out there, is woefully inaccurate for what we are
 doing.

 "Test pass, I'm not sure about all this but I'm going to commit"
 (Tests never pass, must be a lie anyway)
 "Leaving on vacation, going to 

Re: [VOTE] Release Lucene/Solr 8.3.0 RC2

[Shalin Shekhar Mangar]

It seems that the Lucene 8.3.0 release announcement hasn't made its way to
the asf-announce mailing list. I cannot find it in the archives but I see
the solr one there.

On Mon, Nov 4, 2019 at 1:06 AM Jan Høydahl  wrote:

> In my opinion it would have been better to delay the release announcement
> another day in order to collect release-notes feedback, than to rush it out.
> But apology is accepted :) Being RM during such a stressful week is not
> ideal. You did well.
> Looking forward to more detailed feedback regarding ReleaseWizard!
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> 3. nov. 2019 kl. 18:42 skrev Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com>:
>
> according to lucene’s release todo the rm must send a [RESULT][VOTE] email.
>
> You're wrong.
>
> The Lucene's Release TODO page says this:
> "Announce that the vote has passed on the dev mailing list, ideally
> with subject beginning [RESULT]"
>
> ^ I did announce on dev list that the vote passed. It wasn't done
> "ideally" ;-)
>
> On Sun, Nov 3, 2019 at 11:03 PM Ishan Chattopadhyaya
>  wrote:
>
>
> Thanks Jan, for the encouragement and for the release wizard tool.
> All of last week, I was battling several personal emergencies as well as
> professional responsibilities. I was so short on time that I cut some
> corners (in the following two places), *deliberately*.
>
> 1. I checked how often have we sent out a vote results thread in the past,
> and it was only a handful of times; hence I skipped it.
> 2. I had to run the build and publish script at least 4-5 times before the
> stars aligned (tests passed, i was ready at the exact moment GPG prompt
> appears for password etc). That left me so exhausted that i deferred the
> preparation of draft notes for a later moment. Also, after the Moin Moin to
> confluence migration, I spent at least 15-20 minutes to figure out how to
> create a release notes page at the right place. Manually curating the draft
> highlights/notes is the most labour intensive task and I propose some
> automation banking on a Jira field.
>
> Since there was no chance for community to weigh in on the notes before it
> was released, i apologize for any mistakes in doing so. Here they are:
> https://cwiki.apache.org/confluence/display/LUCENE/ReleaseNote83
> https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote83
>
> Regards,
> Ishan
>
> On Sun, 3 Nov, 2019, 2:27 PM Jan Høydahl,  wrote:
>
>
> Ishan,
>
> Congrats with the release!
>
> I may have missed it, but where were the draft release notes for review
> and comments, I was waiting for an invitation...
>
> And how many +1 PMC votes were there? I’m not in doubt that the vote
> passed, but according to lucene’s release todo the rm must send a
> [RESULT][VOTE] email.
>
> Jan Høydahl
>
> 1. nov. 2019 kl. 19:13 skrev Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com>:
>
> Right, didn't start a new thread just for the result of this vote. I
> think such a thread has limited value and seems to be very sparingly
> followed.
> As of now, I'm waiting for mirrors to sync up (I'll wait few hours
> more) and working on curating the changelog, updating the website etc.
> Thanks and regards,
> Ishan
>
> On Fri, Nov 1, 2019 at 9:56 PM Jan Høydahl  wrote:
>
> I have not seen the vote result thread yet?
>
> Jan Høydahl
>
> 30. okt. 2019 kl. 22:46 skrev Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com>:
>
> 
> Thanks to all who voted. The vote has passed. I'll proceed to publish the
> artifacts next.
>
> On Thu, 31 Oct, 2019, 2:44 AM Adrien Grand,  wrote:
>
> Ishan, should we close this vote?
>
> Le ven. 25 oct. 2019 à 20:32, Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com> a écrit :
>
>
> Please vote for release candidate 2 for Lucene/Solr 8.3.0
>
> The artifacts can be downloaded from:
>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.3.0-RC2-rev2aa586909b911e66e1d8863aa89f173d69f86cd2
>
> You can run the smoke tester directly with this command:
>
> python3 -u dev-tools/scripts/smokeTestRelease.py \
>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.3.0-RC2-rev2aa586909b911e66e1d8863aa89f173d69f86cd2
>
> The vote will be open for at least 3 working days, i.e. until
> 2019-10-30 19:00 UTC.
>
> [*] +1  approve
> [ ] +0  no opinion
> [ ] -1  disapprove (and reason why)
>
> Here is my +1
> 
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
> 

Re: SolrCloud is sick.

[Scott Blum]

WHO OVERSEES THE OVERSEER

On Tue, Nov 5, 2019 at 5:16 PM Mark Miller  wrote:

> bq.   SolrCloud is a ballerina. Doesn't look it, cause we dont take care
> of it.
>
> And this is why I'm so devastated by the Overseer. I don't blame anyone
> person. Where was the manual, where was my intervention. I whispered
> Overseer and cut one more thing off my list of responsibilities.
>
> But the overseer is supposed to be so light weight and easy breezy. Giving
> up leader shop at the most signs of trouble, keeping communication small
> and tight with tiny json distrib queue pub/sub updates. Little about stat
> change, hardly needed. Hardly ever talking to Zookeeper.
>
> Our whole system is not moved hard against this, but nothing so much as
> the Overseer. It has very scary, very tricky, custom ZK code. It has major
> communication with ZK. It has little to weak ability to properly throttle
> itself or deal with things intelligently. It's almost a brute force tactic.
> And it clings to being Overseer like a moth to flame. It's designed to be
> on a dedicated hardwar and then mostly to not make any reasonable use of
> that hardware.
>
> I blame me more than anyone for that. I am mad at me. It's just an
> absolute brain bash with a sledge hammer to the system. And i never
> communicated the system very well. I was overloaded.
>
> On Tue, Nov 5, 2019 at 11:01 AM Mark Miller  wrote:
>
>> And now we are to meat of it. Fill in here:
>> https://issues.apache.org/jira/browse/SOLR-13888
>>
>> We can play in a better world, we can have fun, but some of you are going
>> to have to adjust your ways. In the most convenient way possible. You are
>> all great people, I don't want to cause you annoyance, but there are
>> certain requirements to building an aircraft, and there certain
>> requirements to building this.
>>
>> On Tue, Nov 5, 2019 at 10:44 AM Mark Miller 
>> wrote:
>>
>>> If you had any idea how much suffering just that has caused. Not just
>>> users, but us.
>>>
>>> Mark
>>>
>>> On Tue, Nov 5, 2019 at 10:38 AM Mark Miller 
>>> wrote:
>>>
 It’s like 6-7 years since I quickly added a shitty collections API in
 my free time because we desperately needed SOMETHING. I don’t know if I
 tried to make it wait for proper state or what , it was a stub to try get
 things moving. That call, to this day, along with all our other checks,
 until some tests ones recently, is garbage.

 If I downloaded a database, and a lot the time, after the create a
 database call returned, my database was not ready, I’d saw wow. Terrible
 bug got through. If it was a persistent issue for over half a decade? My
 god.

 Look I just spent that half decade upgrading from Solr 4 to whatever. I
 was mostly out of the loop. But this is crazy, me in there too.

 Mark

 On Tue, Nov 5, 2019 at 10:05 AM Mark Miller 
 wrote:

> I'll tell you what guys, development right now sucks. I don't enjoy.
>
> But when I start to put things in shape? I get this smile, and I start
> going with the feeling of I don't need you guys, I don't users, I don't
> need a job, cause just this is figgen nice.
>
> On Tue, Nov 5, 2019 at 9:59 AM Mark Miller 
> wrote:
>
>> I suppose I should toss one more out.
>>
>> Hell yes, we will be using curator.
>>
>> It's insane for any group larger than 2-3 to directly use ZooKeeper.
>> Even for that group, you want some damn good reasons to not use curator. 
>> We
>> can start using more assembly too (joke Yonik).
>>
>> Curator was an option initially. Then it was yet another project
>> hosted by Netflix. Now it is essential.
>>
>>
>> - Mark
>>
>> On Tue, Nov 5, 2019 at 9:41 AM Mark Miller 
>> wrote:
>>
>>> And look, we started pretty deep in the hole. Solr started with tons
>>> of bug or limitations that hardly mattered to it and hit SolrCloud in 
>>> the
>>> eye like a train. And we were not setup to deal with that.
>>>
>>> We never had a nice garden for SolrCloud. We started in a mess,
>>> thinking, eventually we clear the overgrowth, and we are all good. And 
>>> then
>>> we started building our house and that garden went wild with a life of 
>>> it's
>>> own.
>>>
>>> And our development practices, amazingly above many many many groups
>>> and standards out there, is woefully inaccurate for what we are doing.
>>>
>>> "Test pass, I'm not sure about all this but I'm going to commit"
>>> (Tests never pass, must be a lie anyway)
>>> "Leaving on vacation, going to fire this in"
>>> "No one has looked at this huge thing, it's been a while, going to
>>> commit"
>>> *commit*
>>>
>>> And comments to that affect pretty much wrap up our careful and
>>> thoughtful attitude.
>>>
>>> And then of course we come and clean up after, careful gardeners
>>> that we 

Questions about corrupted Segments files.

[Kayak28]

Hello, Community members:

I am using Solr 7.7.2.
On the other day, while indexing to the Solr, my computer powered off.
As a result, there are corrupted segment files.

Is there any way to fix the corrupted segment files without re-indexing?

I have read a blog post (in Japanese) writing about checkIndex method which
can be used to determine/fix corrupted segment files, but when I tried to
run the following command, I got the error message.
So, I am not sure if checkIndex can actually fix the index files.


java -cp lucene-core-7.7.2.jar -ea:org.apache.lucene...
org.apache.lucene.index.CheckIndex solr/server/solr/basic_copy/data/index
-fix


ERROR: unexpected extra argument '-fix'



If anybody knows about either a way to fix corrupted segment files or a way
to use checkIndex '-fix' option correctly, could you please let me know?

Any clue will be very appreciated.

Sincerely,
Kaya Ota

Re: SolrCloud is sick.

[Mark Miller]

 Bottom line, garden is a fricken mess and there are tons of attempts to
solve shit the wrong way cause nobody understands what was intended here.
At least one of those is on me.

Mark

On Tue, Nov 5, 2019 at 10:16 PM Mark Miller  wrote:

> bq.   SolrCloud is a ballerina. Doesn't look it, cause we dont take care
> of it.
>
> And this is why I'm so devastated by the Overseer. I don't blame anyone
> person. Where was the manual, where was my intervention. I whispered
> Overseer and cut one more thing off my list of responsibilities.
>
> But the overseer is supposed to be so light weight and easy breezy. Giving
> up leader shop at the most signs of trouble, keeping communication small
> and tight with tiny json distrib queue pub/sub updates. Little about stat
> change, hardly needed. Hardly ever talking to Zookeeper.
>
> Our whole system is not moved hard against this, but nothing so much as
> the Overseer. It has very scary, very tricky, custom ZK code. It has major
> communication with ZK. It has little to weak ability to properly throttle
> itself or deal with things intelligently. It's almost a brute force tactic.
> And it clings to being Overseer like a moth to flame. It's designed to be
> on a dedicated hardwar and then mostly to not make any reasonable use of
> that hardware.
>
> I blame me more than anyone for that. I am mad at me. It's just an
> absolute brain bash with a sledge hammer to the system. And i never
> communicated the system very well. I was overloaded.
>
> On Tue, Nov 5, 2019 at 11:01 AM Mark Miller  wrote:
>
>> And now we are to meat of it. Fill in here:
>> https://issues.apache.org/jira/browse/SOLR-13888
>>
>> We can play in a better world, we can have fun, but some of you are going
>> to have to adjust your ways. In the most convenient way possible. You are
>> all great people, I don't want to cause you annoyance, but there are
>> certain requirements to building an aircraft, and there certain
>> requirements to building this.
>>
>> On Tue, Nov 5, 2019 at 10:44 AM Mark Miller 
>> wrote:
>>
>>> If you had any idea how much suffering just that has caused. Not just
>>> users, but us.
>>>
>>> Mark
>>>
>>> On Tue, Nov 5, 2019 at 10:38 AM Mark Miller 
>>> wrote:
>>>
 It’s like 6-7 years since I quickly added a shitty collections API in
 my free time because we desperately needed SOMETHING. I don’t know if I
 tried to make it wait for proper state or what , it was a stub to try get
 things moving. That call, to this day, along with all our other checks,
 until some tests ones recently, is garbage.

 If I downloaded a database, and a lot the time, after the create a
 database call returned, my database was not ready, I’d saw wow. Terrible
 bug got through. If it was a persistent issue for over half a decade? My
 god.

 Look I just spent that half decade upgrading from Solr 4 to whatever. I
 was mostly out of the loop. But this is crazy, me in there too.

 Mark

 On Tue, Nov 5, 2019 at 10:05 AM Mark Miller 
 wrote:

> I'll tell you what guys, development right now sucks. I don't enjoy.
>
> But when I start to put things in shape? I get this smile, and I start
> going with the feeling of I don't need you guys, I don't users, I don't
> need a job, cause just this is figgen nice.
>
> On Tue, Nov 5, 2019 at 9:59 AM Mark Miller 
> wrote:
>
>> I suppose I should toss one more out.
>>
>> Hell yes, we will be using curator.
>>
>> It's insane for any group larger than 2-3 to directly use ZooKeeper.
>> Even for that group, you want some damn good reasons to not use curator. 
>> We
>> can start using more assembly too (joke Yonik).
>>
>> Curator was an option initially. Then it was yet another project
>> hosted by Netflix. Now it is essential.
>>
>>
>> - Mark
>>
>> On Tue, Nov 5, 2019 at 9:41 AM Mark Miller 
>> wrote:
>>
>>> And look, we started pretty deep in the hole. Solr started with tons
>>> of bug or limitations that hardly mattered to it and hit SolrCloud in 
>>> the
>>> eye like a train. And we were not setup to deal with that.
>>>
>>> We never had a nice garden for SolrCloud. We started in a mess,
>>> thinking, eventually we clear the overgrowth, and we are all good. And 
>>> then
>>> we started building our house and that garden went wild with a life of 
>>> it's
>>> own.
>>>
>>> And our development practices, amazingly above many many many groups
>>> and standards out there, is woefully inaccurate for what we are doing.
>>>
>>> "Test pass, I'm not sure about all this but I'm going to commit"
>>> (Tests never pass, must be a lie anyway)
>>> "Leaving on vacation, going to fire this in"
>>> "No one has looked at this huge thing, it's been a while, going to
>>> commit"
>>> *commit*
>>>
>>> And comments to that affect pretty 

Re: SolrCloud is sick.

[Mark Miller]

bq.   SolrCloud is a ballerina. Doesn't look it, cause we dont take care of
it.

And this is why I'm so devastated by the Overseer. I don't blame anyone
person. Where was the manual, where was my intervention. I whispered
Overseer and cut one more thing off my list of responsibilities.

But the overseer is supposed to be so light weight and easy breezy. Giving
up leader shop at the most signs of trouble, keeping communication small
and tight with tiny json distrib queue pub/sub updates. Little about stat
change, hardly needed. Hardly ever talking to Zookeeper.

Our whole system is not moved hard against this, but nothing so much as the
Overseer. It has very scary, very tricky, custom ZK code. It has major
communication with ZK. It has little to weak ability to properly throttle
itself or deal with things intelligently. It's almost a brute force tactic.
And it clings to being Overseer like a moth to flame. It's designed to be
on a dedicated hardwar and then mostly to not make any reasonable use of
that hardware.

I blame me more than anyone for that. I am mad at me. It's just an absolute
brain bash with a sledge hammer to the system. And i never communicated the
system very well. I was overloaded.

On Tue, Nov 5, 2019 at 11:01 AM Mark Miller  wrote:

> And now we are to meat of it. Fill in here:
> https://issues.apache.org/jira/browse/SOLR-13888
>
> We can play in a better world, we can have fun, but some of you are going
> to have to adjust your ways. In the most convenient way possible. You are
> all great people, I don't want to cause you annoyance, but there are
> certain requirements to building an aircraft, and there certain
> requirements to building this.
>
> On Tue, Nov 5, 2019 at 10:44 AM Mark Miller  wrote:
>
>> If you had any idea how much suffering just that has caused. Not just
>> users, but us.
>>
>> Mark
>>
>> On Tue, Nov 5, 2019 at 10:38 AM Mark Miller 
>> wrote:
>>
>>> It’s like 6-7 years since I quickly added a shitty collections API in my
>>> free time because we desperately needed SOMETHING. I don’t know if I tried
>>> to make it wait for proper state or what , it was a stub to try get things
>>> moving. That call, to this day, along with all our other checks, until some
>>> tests ones recently, is garbage.
>>>
>>> If I downloaded a database, and a lot the time, after the create a
>>> database call returned, my database was not ready, I’d saw wow. Terrible
>>> bug got through. If it was a persistent issue for over half a decade? My
>>> god.
>>>
>>> Look I just spent that half decade upgrading from Solr 4 to whatever. I
>>> was mostly out of the loop. But this is crazy, me in there too.
>>>
>>> Mark
>>>
>>> On Tue, Nov 5, 2019 at 10:05 AM Mark Miller 
>>> wrote:
>>>
 I'll tell you what guys, development right now sucks. I don't enjoy.

 But when I start to put things in shape? I get this smile, and I start
 going with the feeling of I don't need you guys, I don't users, I don't
 need a job, cause just this is figgen nice.

 On Tue, Nov 5, 2019 at 9:59 AM Mark Miller 
 wrote:

> I suppose I should toss one more out.
>
> Hell yes, we will be using curator.
>
> It's insane for any group larger than 2-3 to directly use ZooKeeper.
> Even for that group, you want some damn good reasons to not use curator. 
> We
> can start using more assembly too (joke Yonik).
>
> Curator was an option initially. Then it was yet another project
> hosted by Netflix. Now it is essential.
>
>
> - Mark
>
> On Tue, Nov 5, 2019 at 9:41 AM Mark Miller 
> wrote:
>
>> And look, we started pretty deep in the hole. Solr started with tons
>> of bug or limitations that hardly mattered to it and hit SolrCloud in the
>> eye like a train. And we were not setup to deal with that.
>>
>> We never had a nice garden for SolrCloud. We started in a mess,
>> thinking, eventually we clear the overgrowth, and we are all good. And 
>> then
>> we started building our house and that garden went wild with a life of 
>> it's
>> own.
>>
>> And our development practices, amazingly above many many many groups
>> and standards out there, is woefully inaccurate for what we are doing.
>>
>> "Test pass, I'm not sure about all this but I'm going to commit"
>> (Tests never pass, must be a lie anyway)
>> "Leaving on vacation, going to fire this in"
>> "No one has looked at this huge thing, it's been a while, going to
>> commit"
>> *commit*
>>
>> And comments to that affect pretty much wrap up our careful and
>> thoughtful attitude.
>>
>> And then of course we come and clean up after, careful gardeners that
>> we are ... no, we don't. We are not setup to be gardeners, we are not
>> trying, even if we do, I only like grass and screw the other plants.
>>
>> Without SolrCloud, Solr wold be in trouble as well. Brute that it is,
>> 

Re: [ANNOUNCE] Apache Solr 8.3.0 released

[Paras Lehana]

Hey Cassandra,

Thank you for prompt response. I look forward to contribute to Solr
Documentation
. Really
love the community! ^_^

PS: We need to fix the Public Servers Listing
 section on homepage that is independent
of the Ref Guide version. There seems to be no way to apply for listing
.

On Tue, 5 Nov 2019 at 20:53, Cassandra Targett 
wrote:

> We’re still working out the changes to the publication process, and got a
> couple wires crossed that prevented the Ref Guide from being published at
> the same time for this release.
>
> I’ve published the final version now:
> http://lucene.apache.org/solr/guide/8_3/.
>
> Apologies for the confusion, by the next release we expect to have all
> this worked out.
> On Nov 4, 2019, 2:30 AM -0600, Paras Lehana ,
> wrote:
> > Hey Ishan,
> >
> > Somedays back it was announced that the foundation will majorly focus on
> > HTML guides now and thus, 8.2 guide was released (which was said to be
> > shorter than 8.0). C*an we expect Ref Guide 8.3 in coming days* or maybe
> is
> > it not required? Asking because we are planning to upgrade to latest
> stable
> > Solr versions and to keep us updated all times, we will be referring the
> > latest guides and incorporated changes in each release.
> >
> > On Sun, 3 Nov 2019 at 04:34, Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com>
> > wrote:
> >
> > > ## 2 November 2019, Apache Solr™ 8.3.0 available
> > >
> > > The Lucene PMC is pleased to announce the release of Apache Solr 8.3.0.
> > >
> > > Solr is the popular, blazing fast, open source NoSQL search platform
> > > from the Apache Lucene project. Its major features include powerful
> > > full-text search, hit highlighting, faceted search, dynamic
> > > clustering, database integration, rich document handling, and
> > > geospatial search. Solr is highly scalable, providing fault tolerant
> > > distributed search and indexing, and powers the search and navigation
> > > features of many of the world's largest internet sites.
> > >
> > > Solr 8.3.0 is available for immediate download at:
> > >
> > > 
> > >
> > > ### Solr 8.3.0 Release Highlights:
> > >
> > > *Two dimensional routed aliases are now available for organizing
> > > collections based on the data values of two fields
> > > *SPLITSHARD implements a new splitByPrefix option that takes into
> > > account the actual document distribution when using compositeIds
> > > *QueryElevationComponent can have query rules configured with
> > > match="subset" wherein the words need only match a subset of the
> > > query's words and in any order
> > > *Command line option to export documents to a file
> > > *Support deterministic replica routing preferences for better cache
> usage
> > > *Ability to query aliases in Solr Admin UI
> > > *JWTAuthPlugin supports multiple JWKS endpoints and multiple IdP
> issuers
> > > *JSON faceting now supports arbitrary ranges for range facets
> > > *Support integral plots, cosine distance and string truncation with
> > > math expressions (Joel Bernstein)
> > > *New cat() stream source to create tuples from lines in local files
> > > *Add upper, lower, trim and split Stream Evaluators
> > > *Add CsvStream, TsvStream Streaming Expressions and supporting
> > > Stream Evaluators
> > > *Add CaffeineCache, an efficient implementation of SolrCache
> > > *Live SPLITSHARD can lose updates due to cluster state change
> > > between checking if the current shard is active and later checking if
> > > there are any sub-shard leaders to forward the update to
> > > *Fix for SPLITSHARD (async) with failures in underlying
> > > sub-operations can result in data loss
> > > *Allow dynamic resizing of SolrCache-s
> > > *Allow optional redaction of data saved by 'bin/solr autoscaling -save'
> > > *Optimized large managed schema modifications (internal O(n^2) problem)
> > > *Max idle time support for SolrCache implementations
> > > *Add Prometheus Exporter GC and Heap options
> > > *SSL: Adding Enabling/Disabling client's hostname verification config
> > > *Introducing SolrClient.ping(collection) in SolrJ
> > > *Fix for CDCR bootstrap not replicating index to the replicas of
> > > target cluster
> > > *Fixed a race condition when initializing metrics for new security
> > > plugins on security.json change
> > > *Fixed JWTAuthPlugin to update metrics prior to continuing w/other
> > > filters or returning error
> > > *Fixed distributed grouping when multiple 'fl' params are specified
> > > *JMX MBeans are not exposed because of race condition between
> > > creating platform mbean server and registering mbeans
> > > *Fix for class-cast issues during atomic-update 'removeregex'
> operations
> > > *Fix for multi-node race condition to create/remove nodeLost markers
> > > *Fix for too many cascading calls to remote 

Re: SolrCloud is sick.

[Mark Miller]

And now we are to meat of it. Fill in here:
https://issues.apache.org/jira/browse/SOLR-13888

We can play in a better world, we can have fun, but some of you are going
to have to adjust your ways. In the most convenient way possible. You are
all great people, I don't want to cause you annoyance, but there are
certain requirements to building an aircraft, and there certain
requirements to building this.

On Tue, Nov 5, 2019 at 10:44 AM Mark Miller  wrote:

> If you had any idea how much suffering just that has caused. Not just
> users, but us.
>
> Mark
>
> On Tue, Nov 5, 2019 at 10:38 AM Mark Miller  wrote:
>
>> It’s like 6-7 years since I quickly added a shitty collections API in my
>> free time because we desperately needed SOMETHING. I don’t know if I tried
>> to make it wait for proper state or what , it was a stub to try get things
>> moving. That call, to this day, along with all our other checks, until some
>> tests ones recently, is garbage.
>>
>> If I downloaded a database, and a lot the time, after the create a
>> database call returned, my database was not ready, I’d saw wow. Terrible
>> bug got through. If it was a persistent issue for over half a decade? My
>> god.
>>
>> Look I just spent that half decade upgrading from Solr 4 to whatever. I
>> was mostly out of the loop. But this is crazy, me in there too.
>>
>> Mark
>>
>> On Tue, Nov 5, 2019 at 10:05 AM Mark Miller 
>> wrote:
>>
>>> I'll tell you what guys, development right now sucks. I don't enjoy.
>>>
>>> But when I start to put things in shape? I get this smile, and I start
>>> going with the feeling of I don't need you guys, I don't users, I don't
>>> need a job, cause just this is figgen nice.
>>>
>>> On Tue, Nov 5, 2019 at 9:59 AM Mark Miller 
>>> wrote:
>>>
 I suppose I should toss one more out.

 Hell yes, we will be using curator.

 It's insane for any group larger than 2-3 to directly use ZooKeeper.
 Even for that group, you want some damn good reasons to not use curator. We
 can start using more assembly too (joke Yonik).

 Curator was an option initially. Then it was yet another project hosted
 by Netflix. Now it is essential.


 - Mark

 On Tue, Nov 5, 2019 at 9:41 AM Mark Miller 
 wrote:

> And look, we started pretty deep in the hole. Solr started with tons
> of bug or limitations that hardly mattered to it and hit SolrCloud in the
> eye like a train. And we were not setup to deal with that.
>
> We never had a nice garden for SolrCloud. We started in a mess,
> thinking, eventually we clear the overgrowth, and we are all good. And 
> then
> we started building our house and that garden went wild with a life of 
> it's
> own.
>
> And our development practices, amazingly above many many many groups
> and standards out there, is woefully inaccurate for what we are doing.
>
> "Test pass, I'm not sure about all this but I'm going to commit"
> (Tests never pass, must be a lie anyway)
> "Leaving on vacation, going to fire this in"
> "No one has looked at this huge thing, it's been a while, going to
> commit"
> *commit*
>
> And comments to that affect pretty much wrap up our careful and
> thoughtful attitude.
>
> And then of course we come and clean up after, careful gardeners that
> we are ... no, we don't. We are not setup to be gardeners, we are not
> trying, even if we do, I only like grass and screw the other plants.
>
> Without SolrCloud, Solr wold be in trouble as well. Brute that it is,
> it could go a few more rounds. SolrCloud is a ballerina. Doesn't look it,
> cause we dont take care of it. But it is, and it cannot take the beating
> that the brute does.
>
> - Mark
>
> On Tue, Nov 5, 2019 at 5:19 AM Mark Miller 
> wrote:
>
>> Basically I can fix 99% of this without you guys - with simple care
>> and effort and time that non of you are likely in the circumstances of
>> being able to duplicate.. Been there done that, made it 100x-1000x faster
>> to boot and added all kinds of fun.
>>
>> But I can't build the rest of Solr. I don't care about facets. So
>> let's meet half way.
>>
>> On Tue, Nov 5, 2019 at 5:14 AM Mark Miller 
>> wrote:
>>
>>> There are 10,000 problems here.
>>>
>>> So if you eventually land on one possible solution you agree on, we
>>> a little closer.
>>>
>>> There is no problem with the current design. Design's can always be
>>> improved, sure. I've made this one fast. You won't believe me fast. The 
>>> low
>>> hanging fruit is astronomical, there is more fruit above that.
>>>
>>> We never focused on performance. Or at least didn't. That's after we
>>> harden.
>>>
>>> Except performance is the key to everything.
>>>
>>> SolrCloud is not the only problem. The design of Solr, of 

Re: SolrCloud is sick.

[Mark Miller]

If you had any idea how much suffering just that has caused. Not just
users, but us.

Mark

On Tue, Nov 5, 2019 at 10:38 AM Mark Miller  wrote:

> It’s like 6-7 years since I quickly added a shitty collections API in my
> free time because we desperately needed SOMETHING. I don’t know if I tried
> to make it wait for proper state or what , it was a stub to try get things
> moving. That call, to this day, along with all our other checks, until some
> tests ones recently, is garbage.
>
> If I downloaded a database, and a lot the time, after the create a
> database call returned, my database was not ready, I’d saw wow. Terrible
> bug got through. If it was a persistent issue for over half a decade? My
> god.
>
> Look I just spent that half decade upgrading from Solr 4 to whatever. I
> was mostly out of the loop. But this is crazy, me in there too.
>
> Mark
>
> On Tue, Nov 5, 2019 at 10:05 AM Mark Miller  wrote:
>
>> I'll tell you what guys, development right now sucks. I don't enjoy.
>>
>> But when I start to put things in shape? I get this smile, and I start
>> going with the feeling of I don't need you guys, I don't users, I don't
>> need a job, cause just this is figgen nice.
>>
>> On Tue, Nov 5, 2019 at 9:59 AM Mark Miller  wrote:
>>
>>> I suppose I should toss one more out.
>>>
>>> Hell yes, we will be using curator.
>>>
>>> It's insane for any group larger than 2-3 to directly use ZooKeeper.
>>> Even for that group, you want some damn good reasons to not use curator. We
>>> can start using more assembly too (joke Yonik).
>>>
>>> Curator was an option initially. Then it was yet another project hosted
>>> by Netflix. Now it is essential.
>>>
>>>
>>> - Mark
>>>
>>> On Tue, Nov 5, 2019 at 9:41 AM Mark Miller 
>>> wrote:
>>>
 And look, we started pretty deep in the hole. Solr started with tons of
 bug or limitations that hardly mattered to it and hit SolrCloud in the eye
 like a train. And we were not setup to deal with that.

 We never had a nice garden for SolrCloud. We started in a mess,
 thinking, eventually we clear the overgrowth, and we are all good. And then
 we started building our house and that garden went wild with a life of it's
 own.

 And our development practices, amazingly above many many many groups
 and standards out there, is woefully inaccurate for what we are doing.

 "Test pass, I'm not sure about all this but I'm going to commit" (Tests
 never pass, must be a lie anyway)
 "Leaving on vacation, going to fire this in"
 "No one has looked at this huge thing, it's been a while, going to
 commit"
 *commit*

 And comments to that affect pretty much wrap up our careful and
 thoughtful attitude.

 And then of course we come and clean up after, careful gardeners that
 we are ... no, we don't. We are not setup to be gardeners, we are not
 trying, even if we do, I only like grass and screw the other plants.

 Without SolrCloud, Solr wold be in trouble as well. Brute that it is,
 it could go a few more rounds. SolrCloud is a ballerina. Doesn't look it,
 cause we dont take care of it. But it is, and it cannot take the beating
 that the brute does.

 - Mark

 On Tue, Nov 5, 2019 at 5:19 AM Mark Miller 
 wrote:

> Basically I can fix 99% of this without you guys - with simple care
> and effort and time that non of you are likely in the circumstances of
> being able to duplicate.. Been there done that, made it 100x-1000x faster
> to boot and added all kinds of fun.
>
> But I can't build the rest of Solr. I don't care about facets. So
> let's meet half way.
>
> On Tue, Nov 5, 2019 at 5:14 AM Mark Miller 
> wrote:
>
>> There are 10,000 problems here.
>>
>> So if you eventually land on one possible solution you agree on, we a
>> little closer.
>>
>> There is no problem with the current design. Design's can always be
>> improved, sure. I've made this one fast. You won't believe me fast. The 
>> low
>> hanging fruit is astronomical, there is more fruit above that.
>>
>> We never focused on performance. Or at least didn't. That's after we
>> harden.
>>
>> Except performance is the key to everything.
>>
>> SolrCloud is not the only problem. The design of Solr, of SolrCloud,
>> they are fine. Change them, I don't care. Later. They are not a problem.
>>
>> But Solr has as many problems as SolrCloud at this point. This just
>> mater  a whole hell of lot less unless they are messing with SolrCloud.
>> Standalone is more of a brute.
>>
>> We have 60 modules that are interconnected. We have a huge code base.
>> That is also fine.
>>
>> We don't tend our garden. That's not fine. I've tended the garden
>> before without one - more than once before. It's a great damn garden. You
>> guys only get to see it grown over and 

Re: SolrCloud is sick.

[Mark Miller]

It’s like 6-7 years since I quickly added a shitty collections API in my
free time because we desperately needed SOMETHING. I don’t know if I tried
to make it wait for proper state or what , it was a stub to try get things
moving. That call, to this day, along with all our other checks, until some
tests ones recently, is garbage.

If I downloaded a database, and a lot the time, after the create a database
call returned, my database was not ready, I’d saw wow. Terrible bug got
through. If it was a persistent issue for over half a decade? My god.

Look I just spent that half decade upgrading from Solr 4 to whatever. I was
mostly out of the loop. But this is crazy, me in there too.

Mark

On Tue, Nov 5, 2019 at 10:05 AM Mark Miller  wrote:

> I'll tell you what guys, development right now sucks. I don't enjoy.
>
> But when I start to put things in shape? I get this smile, and I start
> going with the feeling of I don't need you guys, I don't users, I don't
> need a job, cause just this is figgen nice.
>
> On Tue, Nov 5, 2019 at 9:59 AM Mark Miller  wrote:
>
>> I suppose I should toss one more out.
>>
>> Hell yes, we will be using curator.
>>
>> It's insane for any group larger than 2-3 to directly use ZooKeeper. Even
>> for that group, you want some damn good reasons to not use curator. We can
>> start using more assembly too (joke Yonik).
>>
>> Curator was an option initially. Then it was yet another project hosted
>> by Netflix. Now it is essential.
>>
>>
>> - Mark
>>
>> On Tue, Nov 5, 2019 at 9:41 AM Mark Miller  wrote:
>>
>>> And look, we started pretty deep in the hole. Solr started with tons of
>>> bug or limitations that hardly mattered to it and hit SolrCloud in the eye
>>> like a train. And we were not setup to deal with that.
>>>
>>> We never had a nice garden for SolrCloud. We started in a mess,
>>> thinking, eventually we clear the overgrowth, and we are all good. And then
>>> we started building our house and that garden went wild with a life of it's
>>> own.
>>>
>>> And our development practices, amazingly above many many many groups and
>>> standards out there, is woefully inaccurate for what we are doing.
>>>
>>> "Test pass, I'm not sure about all this but I'm going to commit" (Tests
>>> never pass, must be a lie anyway)
>>> "Leaving on vacation, going to fire this in"
>>> "No one has looked at this huge thing, it's been a while, going to
>>> commit"
>>> *commit*
>>>
>>> And comments to that affect pretty much wrap up our careful and
>>> thoughtful attitude.
>>>
>>> And then of course we come and clean up after, careful gardeners that we
>>> are ... no, we don't. We are not setup to be gardeners, we are not trying,
>>> even if we do, I only like grass and screw the other plants.
>>>
>>> Without SolrCloud, Solr wold be in trouble as well. Brute that it is, it
>>> could go a few more rounds. SolrCloud is a ballerina. Doesn't look it,
>>> cause we dont take care of it. But it is, and it cannot take the beating
>>> that the brute does.
>>>
>>> - Mark
>>>
>>> On Tue, Nov 5, 2019 at 5:19 AM Mark Miller 
>>> wrote:
>>>
 Basically I can fix 99% of this without you guys - with simple care and
 effort and time that non of you are likely in the circumstances of being
 able to duplicate.. Been there done that, made it 100x-1000x faster to boot
 and added all kinds of fun.

 But I can't build the rest of Solr. I don't care about facets. So let's
 meet half way.

 On Tue, Nov 5, 2019 at 5:14 AM Mark Miller 
 wrote:

> There are 10,000 problems here.
>
> So if you eventually land on one possible solution you agree on, we a
> little closer.
>
> There is no problem with the current design. Design's can always be
> improved, sure. I've made this one fast. You won't believe me fast. The 
> low
> hanging fruit is astronomical, there is more fruit above that.
>
> We never focused on performance. Or at least didn't. That's after we
> harden.
>
> Except performance is the key to everything.
>
> SolrCloud is not the only problem. The design of Solr, of SolrCloud,
> they are fine. Change them, I don't care. Later. They are not a problem.
>
> But Solr has as many problems as SolrCloud at this point. This just
> mater  a whole hell of lot less unless they are messing with SolrCloud.
> Standalone is more of a brute.
>
> We have 60 modules that are interconnected. We have a huge code base.
> That is also fine.
>
> We don't tend our garden. That's not fine. I've tended the garden
> before without one - more than once before. It's a great damn garden. You
> guys only get to see it grown over and full of weeds.
>
> Anyway, no redesign, no library, no nothing like that gonna save this.
>
> This is hardly concrete awareness of a problem here. The awareness to
> figure out what actually are the problems and what must be done - that's
> expensive shit 

Re: SolrCloud is sick.

[Mark Miller]

I'll tell you what guys, development right now sucks. I don't enjoy.

But when I start to put things in shape? I get this smile, and I start
going with the feeling of I don't need you guys, I don't users, I don't
need a job, cause just this is figgen nice.

On Tue, Nov 5, 2019 at 9:59 AM Mark Miller  wrote:

> I suppose I should toss one more out.
>
> Hell yes, we will be using curator.
>
> It's insane for any group larger than 2-3 to directly use ZooKeeper. Even
> for that group, you want some damn good reasons to not use curator. We can
> start using more assembly too (joke Yonik).
>
> Curator was an option initially. Then it was yet another project hosted by
> Netflix. Now it is essential.
>
>
> - Mark
>
> On Tue, Nov 5, 2019 at 9:41 AM Mark Miller  wrote:
>
>> And look, we started pretty deep in the hole. Solr started with tons of
>> bug or limitations that hardly mattered to it and hit SolrCloud in the eye
>> like a train. And we were not setup to deal with that.
>>
>> We never had a nice garden for SolrCloud. We started in a mess, thinking,
>> eventually we clear the overgrowth, and we are all good. And then we
>> started building our house and that garden went wild with a life of it's
>> own.
>>
>> And our development practices, amazingly above many many many groups and
>> standards out there, is woefully inaccurate for what we are doing.
>>
>> "Test pass, I'm not sure about all this but I'm going to commit" (Tests
>> never pass, must be a lie anyway)
>> "Leaving on vacation, going to fire this in"
>> "No one has looked at this huge thing, it's been a while, going to commit"
>> *commit*
>>
>> And comments to that affect pretty much wrap up our careful and
>> thoughtful attitude.
>>
>> And then of course we come and clean up after, careful gardeners that we
>> are ... no, we don't. We are not setup to be gardeners, we are not trying,
>> even if we do, I only like grass and screw the other plants.
>>
>> Without SolrCloud, Solr wold be in trouble as well. Brute that it is, it
>> could go a few more rounds. SolrCloud is a ballerina. Doesn't look it,
>> cause we dont take care of it. But it is, and it cannot take the beating
>> that the brute does.
>>
>> - Mark
>>
>> On Tue, Nov 5, 2019 at 5:19 AM Mark Miller  wrote:
>>
>>> Basically I can fix 99% of this without you guys - with simple care and
>>> effort and time that non of you are likely in the circumstances of being
>>> able to duplicate.. Been there done that, made it 100x-1000x faster to boot
>>> and added all kinds of fun.
>>>
>>> But I can't build the rest of Solr. I don't care about facets. So let's
>>> meet half way.
>>>
>>> On Tue, Nov 5, 2019 at 5:14 AM Mark Miller 
>>> wrote:
>>>
 There are 10,000 problems here.

 So if you eventually land on one possible solution you agree on, we a
 little closer.

 There is no problem with the current design. Design's can always be
 improved, sure. I've made this one fast. You won't believe me fast. The low
 hanging fruit is astronomical, there is more fruit above that.

 We never focused on performance. Or at least didn't. That's after we
 harden.

 Except performance is the key to everything.

 SolrCloud is not the only problem. The design of Solr, of SolrCloud,
 they are fine. Change them, I don't care. Later. They are not a problem.

 But Solr has as many problems as SolrCloud at this point. This just
 mater  a whole hell of lot less unless they are messing with SolrCloud.
 Standalone is more of a brute.

 We have 60 modules that are interconnected. We have a huge code base.
 That is also fine.

 We don't tend our garden. That's not fine. I've tended the garden
 before without one - more than once before. It's a great damn garden. You
 guys only get to see it grown over and full of weeds.

 Anyway, no redesign, no library, no nothing like that gonna save this.

 This is hardly concrete awareness of a problem here. The awareness to
 figure out what actually are the problems and what must be done - that's
 expensive shit these days if you ask me. I've been wrong lots tough.






 On Mon, Nov 4, 2019 at 2:26 PM Jörn Franke 
 wrote:

> I guess this is also a bit normal with software that grows over the
> years.
> One could also say that one writes the current use cases and
> interesting future use cases for Solr in a document and designs from
> scratch new - taking only the good pieces out of the existing software.
> Of course there is a certain amount of time where you need to maintain
> both - but this will be also the case for a major rewrite.
>
> > Am 04.11.2019 um 20:58 schrieb Erick Erickson <
> erickerick...@gmail.com>:
> >
> > If Curator would make that easier and we’re doing major surgery
> anyway….
> >
> > But yeah, a nifty, new, more modern tool isn’t going to 

Re: SolrCloud is sick.

[Mark Miller]

I suppose I should toss one more out.

Hell yes, we will be using curator.

It's insane for any group larger than 2-3 to directly use ZooKeeper. Even
for that group, you want some damn good reasons to not use curator. We can
start using more assembly too (joke Yonik).

Curator was an option initially. Then it was yet another project hosted by
Netflix. Now it is essential.


- Mark

On Tue, Nov 5, 2019 at 9:41 AM Mark Miller  wrote:

> And look, we started pretty deep in the hole. Solr started with tons of
> bug or limitations that hardly mattered to it and hit SolrCloud in the eye
> like a train. And we were not setup to deal with that.
>
> We never had a nice garden for SolrCloud. We started in a mess, thinking,
> eventually we clear the overgrowth, and we are all good. And then we
> started building our house and that garden went wild with a life of it's
> own.
>
> And our development practices, amazingly above many many many groups and
> standards out there, is woefully inaccurate for what we are doing.
>
> "Test pass, I'm not sure about all this but I'm going to commit" (Tests
> never pass, must be a lie anyway)
> "Leaving on vacation, going to fire this in"
> "No one has looked at this huge thing, it's been a while, going to commit"
> *commit*
>
> And comments to that affect pretty much wrap up our careful and thoughtful
> attitude.
>
> And then of course we come and clean up after, careful gardeners that we
> are ... no, we don't. We are not setup to be gardeners, we are not trying,
> even if we do, I only like grass and screw the other plants.
>
> Without SolrCloud, Solr wold be in trouble as well. Brute that it is, it
> could go a few more rounds. SolrCloud is a ballerina. Doesn't look it,
> cause we dont take care of it. But it is, and it cannot take the beating
> that the brute does.
>
> - Mark
>
> On Tue, Nov 5, 2019 at 5:19 AM Mark Miller  wrote:
>
>> Basically I can fix 99% of this without you guys - with simple care and
>> effort and time that non of you are likely in the circumstances of being
>> able to duplicate.. Been there done that, made it 100x-1000x faster to boot
>> and added all kinds of fun.
>>
>> But I can't build the rest of Solr. I don't care about facets. So let's
>> meet half way.
>>
>> On Tue, Nov 5, 2019 at 5:14 AM Mark Miller  wrote:
>>
>>> There are 10,000 problems here.
>>>
>>> So if you eventually land on one possible solution you agree on, we a
>>> little closer.
>>>
>>> There is no problem with the current design. Design's can always be
>>> improved, sure. I've made this one fast. You won't believe me fast. The low
>>> hanging fruit is astronomical, there is more fruit above that.
>>>
>>> We never focused on performance. Or at least didn't. That's after we
>>> harden.
>>>
>>> Except performance is the key to everything.
>>>
>>> SolrCloud is not the only problem. The design of Solr, of SolrCloud,
>>> they are fine. Change them, I don't care. Later. They are not a problem.
>>>
>>> But Solr has as many problems as SolrCloud at this point. This just
>>> mater  a whole hell of lot less unless they are messing with SolrCloud.
>>> Standalone is more of a brute.
>>>
>>> We have 60 modules that are interconnected. We have a huge code base.
>>> That is also fine.
>>>
>>> We don't tend our garden. That's not fine. I've tended the garden before
>>> without one - more than once before. It's a great damn garden. You guys
>>> only get to see it grown over and full of weeds.
>>>
>>> Anyway, no redesign, no library, no nothing like that gonna save this.
>>>
>>> This is hardly concrete awareness of a problem here. The awareness to
>>> figure out what actually are the problems and what must be done - that's
>>> expensive shit these days if you ask me. I've been wrong lots tough.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Nov 4, 2019 at 2:26 PM Jörn Franke  wrote:
>>>
 I guess this is also a bit normal with software that grows over the
 years.
 One could also say that one writes the current use cases and
 interesting future use cases for Solr in a document and designs from
 scratch new - taking only the good pieces out of the existing software.
 Of course there is a certain amount of time where you need to maintain
 both - but this will be also the case for a major rewrite.

 > Am 04.11.2019 um 20:58 schrieb Erick Erickson <
 erickerick...@gmail.com>:
 >
 > If Curator would make that easier and we’re doing major surgery
 anyway….
 >
 > But yeah, a nifty, new, more modern tool isn’t going to magically
 help if the design is flawed.
 >
 > Or, if I’m putting my philosophical hat on, code doesn’t get gnarly
 intentionally. It gets gnarly because there are a bunch of problems to be
 solved and you don’t know what they are until you run into them. And it’s
 always a tension between fixing it enough to get by and fixing it by
 refactoring/redesign.
 >
 > But eventually “fixing it enough to get by” totters 

Re: SolrCloud is sick.

[Mark Miller]

And look, we started pretty deep in the hole. Solr started with tons of bug
or limitations that hardly mattered to it and hit SolrCloud in the eye like
a train. And we were not setup to deal with that.

We never had a nice garden for SolrCloud. We started in a mess, thinking,
eventually we clear the overgrowth, and we are all good. And then we
started building our house and that garden went wild with a life of it's
own.

And our development practices, amazingly above many many many groups and
standards out there, is woefully inaccurate for what we are doing.

"Test pass, I'm not sure about all this but I'm going to commit" (Tests
never pass, must be a lie anyway)
"Leaving on vacation, going to fire this in"
"No one has looked at this huge thing, it's been a while, going to commit"
*commit*

And comments to that affect pretty much wrap up our careful and thoughtful
attitude.

And then of course we come and clean up after, careful gardeners that we
are ... no, we don't. We are not setup to be gardeners, we are not trying,
even if we do, I only like grass and screw the other plants.

Without SolrCloud, Solr wold be in trouble as well. Brute that it is, it
could go a few more rounds. SolrCloud is a ballerina. Doesn't look it,
cause we dont take care of it. But it is, and it cannot take the beating
that the brute does.

- Mark

On Tue, Nov 5, 2019 at 5:19 AM Mark Miller  wrote:

> Basically I can fix 99% of this without you guys - with simple care and
> effort and time that non of you are likely in the circumstances of being
> able to duplicate.. Been there done that, made it 100x-1000x faster to boot
> and added all kinds of fun.
>
> But I can't build the rest of Solr. I don't care about facets. So let's
> meet half way.
>
> On Tue, Nov 5, 2019 at 5:14 AM Mark Miller  wrote:
>
>> There are 10,000 problems here.
>>
>> So if you eventually land on one possible solution you agree on, we a
>> little closer.
>>
>> There is no problem with the current design. Design's can always be
>> improved, sure. I've made this one fast. You won't believe me fast. The low
>> hanging fruit is astronomical, there is more fruit above that.
>>
>> We never focused on performance. Or at least didn't. That's after we
>> harden.
>>
>> Except performance is the key to everything.
>>
>> SolrCloud is not the only problem. The design of Solr, of SolrCloud, they
>> are fine. Change them, I don't care. Later. They are not a problem.
>>
>> But Solr has as many problems as SolrCloud at this point. This just
>> mater  a whole hell of lot less unless they are messing with SolrCloud.
>> Standalone is more of a brute.
>>
>> We have 60 modules that are interconnected. We have a huge code base.
>> That is also fine.
>>
>> We don't tend our garden. That's not fine. I've tended the garden before
>> without one - more than once before. It's a great damn garden. You guys
>> only get to see it grown over and full of weeds.
>>
>> Anyway, no redesign, no library, no nothing like that gonna save this.
>>
>> This is hardly concrete awareness of a problem here. The awareness to
>> figure out what actually are the problems and what must be done - that's
>> expensive shit these days if you ask me. I've been wrong lots tough.
>>
>>
>>
>>
>>
>>
>> On Mon, Nov 4, 2019 at 2:26 PM Jörn Franke  wrote:
>>
>>> I guess this is also a bit normal with software that grows over the
>>> years.
>>> One could also say that one writes the current use cases and interesting
>>> future use cases for Solr in a document and designs from scratch new -
>>> taking only the good pieces out of the existing software.
>>> Of course there is a certain amount of time where you need to maintain
>>> both - but this will be also the case for a major rewrite.
>>>
>>> > Am 04.11.2019 um 20:58 schrieb Erick Erickson >> >:
>>> >
>>> > If Curator would make that easier and we’re doing major surgery
>>> anyway….
>>> >
>>> > But yeah, a nifty, new, more modern tool isn’t going to magically help
>>> if the design is flawed.
>>> >
>>> > Or, if I’m putting my philosophical hat on, code doesn’t get gnarly
>>> intentionally. It gets gnarly because there are a bunch of problems to be
>>> solved and you don’t know what they are until you run into them. And it’s
>>> always a tension between fixing it enough to get by and fixing it by
>>> refactoring/redesign.
>>> >
>>> > But eventually “fixing it enough to get by” totters under it’s own
>>> weight and becomes increasingly fragile and you must take the hit and redo
>>> major portions of it. The questions now are:
>>> > 1> are we at that point?
>>> > 2> are we going to put the effort into rewriting some of the worst
>>> offenders?
>>> >
>>> >
>>> >
>>> >> On Nov 4, 2019, at 1:28 PM, Scott Blum  wrote:
>>> >>
>>> >> Figuring out a better overall algorithmic & data structure design
>>> that's an order of magnitude improvement seems far more important than
>>> swapping out libraries.  And I say this as a Curator fan and committer. ;)
>>> >>
>>> >> On Mon, Nov 4, 

Re: [ANNOUNCE] Apache Solr 8.3.0 released

[Cassandra Targett]

We’re still working out the changes to the publication process, and got a 
couple wires crossed that prevented the Ref Guide from being published at the 
same time for this release.

I’ve published the final version now: http://lucene.apache.org/solr/guide/8_3/.

Apologies for the confusion, by the next release we expect to have all this 
worked out.
On Nov 4, 2019, 2:30 AM -0600, Paras Lehana , wrote:
> Hey Ishan,
>
> Somedays back it was announced that the foundation will majorly focus on
> HTML guides now and thus, 8.2 guide was released (which was said to be
> shorter than 8.0). C*an we expect Ref Guide 8.3 in coming days* or maybe is
> it not required? Asking because we are planning to upgrade to latest stable
> Solr versions and to keep us updated all times, we will be referring the
> latest guides and incorporated changes in each release.
>
> On Sun, 3 Nov 2019 at 04:34, Ishan Chattopadhyaya 
> wrote:
>
> > ## 2 November 2019, Apache Solr™ 8.3.0 available
> >
> > The Lucene PMC is pleased to announce the release of Apache Solr 8.3.0.
> >
> > Solr is the popular, blazing fast, open source NoSQL search platform
> > from the Apache Lucene project. Its major features include powerful
> > full-text search, hit highlighting, faceted search, dynamic
> > clustering, database integration, rich document handling, and
> > geospatial search. Solr is highly scalable, providing fault tolerant
> > distributed search and indexing, and powers the search and navigation
> > features of many of the world's largest internet sites.
> >
> > Solr 8.3.0 is available for immediate download at:
> >
> > 
> >
> > ### Solr 8.3.0 Release Highlights:
> >
> > *Two dimensional routed aliases are now available for organizing
> > collections based on the data values of two fields
> > *SPLITSHARD implements a new splitByPrefix option that takes into
> > account the actual document distribution when using compositeIds
> > *QueryElevationComponent can have query rules configured with
> > match="subset" wherein the words need only match a subset of the
> > query's words and in any order
> > *Command line option to export documents to a file
> > *Support deterministic replica routing preferences for better cache usage
> > *Ability to query aliases in Solr Admin UI
> > *JWTAuthPlugin supports multiple JWKS endpoints and multiple IdP issuers
> > *JSON faceting now supports arbitrary ranges for range facets
> > *Support integral plots, cosine distance and string truncation with
> > math expressions (Joel Bernstein)
> > *New cat() stream source to create tuples from lines in local files
> > *Add upper, lower, trim and split Stream Evaluators
> > *Add CsvStream, TsvStream Streaming Expressions and supporting
> > Stream Evaluators
> > *Add CaffeineCache, an efficient implementation of SolrCache
> > *Live SPLITSHARD can lose updates due to cluster state change
> > between checking if the current shard is active and later checking if
> > there are any sub-shard leaders to forward the update to
> > *Fix for SPLITSHARD (async) with failures in underlying
> > sub-operations can result in data loss
> > *Allow dynamic resizing of SolrCache-s
> > *Allow optional redaction of data saved by 'bin/solr autoscaling -save'
> > *Optimized large managed schema modifications (internal O(n^2) problem)
> > *Max idle time support for SolrCache implementations
> > *Add Prometheus Exporter GC and Heap options
> > *SSL: Adding Enabling/Disabling client's hostname verification config
> > *Introducing SolrClient.ping(collection) in SolrJ
> > *Fix for CDCR bootstrap not replicating index to the replicas of
> > target cluster
> > *Fixed a race condition when initializing metrics for new security
> > plugins on security.json change
> > *Fixed JWTAuthPlugin to update metrics prior to continuing w/other
> > filters or returning error
> > *Fixed distributed grouping when multiple 'fl' params are specified
> > *JMX MBeans are not exposed because of race condition between
> > creating platform mbean server and registering mbeans
> > *Fix for class-cast issues during atomic-update 'removeregex' operations
> > *Fix for multi-node race condition to create/remove nodeLost markers
> > *Fix for too many cascading calls to remote servers, which can bring
> > down nodes
> > *Fix for MOVEREPLICA ignoring replica type and always adding 'nrt'
> > replicas
> > *Fix: DistributedZkUpdateProcessor should propagate URP.finish()
> > lifecycle (regression since 8.1)
> >
> >
> > Please read CHANGES.txt for a full list of new features and changes:
> >
> > 
> >
> > Solr 8.3.0 also includes features, optimizations and bugfixes in the
> > corresponding Apache Lucene release:
> >
> > 
> >
> >
> > Note: The Apache Software Foundation uses an extensive mirroring network
> > for
> > distributing releases. It is possible that the mirror you 

Re: SolrCloud is sick.

[Mark Miller]

Basically I can fix 99% of this without you guys - with simple care and
effort and time that non of you are likely in the circumstances of being
able to duplicate.. Been there done that, made it 100x-1000x faster to boot
and added all kinds of fun.

But I can't build the rest of Solr. I don't care about facets. So let's
meet half way.

On Tue, Nov 5, 2019 at 5:14 AM Mark Miller  wrote:

> There are 10,000 problems here.
>
> So if you eventually land on one possible solution you agree on, we a
> little closer.
>
> There is no problem with the current design. Design's can always be
> improved, sure. I've made this one fast. You won't believe me fast. The low
> hanging fruit is astronomical, there is more fruit above that.
>
> We never focused on performance. Or at least didn't. That's after we
> harden.
>
> Except performance is the key to everything.
>
> SolrCloud is not the only problem. The design of Solr, of SolrCloud, they
> are fine. Change them, I don't care. Later. They are not a problem.
>
> But Solr has as many problems as SolrCloud at this point. This just mater
> a whole hell of lot less unless they are messing with SolrCloud. Standalone
> is more of a brute.
>
> We have 60 modules that are interconnected. We have a huge code base. That
> is also fine.
>
> We don't tend our garden. That's not fine. I've tended the garden before
> without one - more than once before. It's a great damn garden. You guys
> only get to see it grown over and full of weeds.
>
> Anyway, no redesign, no library, no nothing like that gonna save this.
>
> This is hardly concrete awareness of a problem here. The awareness to
> figure out what actually are the problems and what must be done - that's
> expensive shit these days if you ask me. I've been wrong lots tough.
>
>
>
>
>
>
> On Mon, Nov 4, 2019 at 2:26 PM Jörn Franke  wrote:
>
>> I guess this is also a bit normal with software that grows over the
>> years.
>> One could also say that one writes the current use cases and interesting
>> future use cases for Solr in a document and designs from scratch new -
>> taking only the good pieces out of the existing software.
>> Of course there is a certain amount of time where you need to maintain
>> both - but this will be also the case for a major rewrite.
>>
>> > Am 04.11.2019 um 20:58 schrieb Erick Erickson > >:
>> >
>> > If Curator would make that easier and we’re doing major surgery
>> anyway….
>> >
>> > But yeah, a nifty, new, more modern tool isn’t going to magically help
>> if the design is flawed.
>> >
>> > Or, if I’m putting my philosophical hat on, code doesn’t get gnarly
>> intentionally. It gets gnarly because there are a bunch of problems to be
>> solved and you don’t know what they are until you run into them. And it’s
>> always a tension between fixing it enough to get by and fixing it by
>> refactoring/redesign.
>> >
>> > But eventually “fixing it enough to get by” totters under it’s own
>> weight and becomes increasingly fragile and you must take the hit and redo
>> major portions of it. The questions now are:
>> > 1> are we at that point?
>> > 2> are we going to put the effort into rewriting some of the worst
>> offenders?
>> >
>> >
>> >
>> >> On Nov 4, 2019, at 1:28 PM, Scott Blum  wrote:
>> >>
>> >> Figuring out a better overall algorithmic & data structure design
>> that's an order of magnitude improvement seems far more important than
>> swapping out libraries.  And I say this as a Curator fan and committer. ;)
>> >>
>> >> On Mon, Nov 4, 2019 at 11:44 AM Erick Erickson <
>> erickerick...@gmail.com> wrote:
>> >> Bram:
>> >>
>> >> Using Curator has been proposed before. It would require significant
>> refactoring b/c of how deeply entwined raw ZK is in the code. That said, if
>> we’re going to do major surgery it may be the right time to consider it.
>> >>
>> >> Erick
>> >>
>>  On Nov 4, 2019, at 9:24 AM, Bram Van Dam 
>> wrote:
>> >>>
>>  SolrCloud is sick right now. The way low level Zookeeper is handeled
>> >>>
>> >>> On an unrelated project, I've stopped using "raw" ZK client access and
>> >>> have switched to Curator. The API is a fair bit easier to work with,
>> and
>> >>> it results in less ugly code. I realize that this won't go very far in
>> >>> resolving more fundamental issues, but it might be something that can
>> >>> help improve the shape of the code.
>> >>>
>> >>> - Bram
>> >>>
>> >>> -
>> >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> >>> For additional commands, e-mail: dev-h...@lucene.apache.org
>> >>>
>> >>
>> >>
>> >> -
>> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> >> For additional commands, e-mail: dev-h...@lucene.apache.org
>> >>
>> >
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> > For additional 

Re: dtd anomaly

[Martin Gainty]

if i delete the invalid host DTD
ENTITY_TermQuery.xml from xslt input folder everything works

thanks uwe

From: Uwe Schindler 
Sent: Monday, November 4, 2019 8:04 AM
To: dev@lucene.apache.org 
Subject: Re: dtd anomaly

This job is ran on every ant build on Jenkins so there is no problem. This task 
you mentioned does not even read that xml file, unless it's somehow misplaced 
in your build directory and detected as build.xml.

We know that there is a bug with ant 1.10 but that was something else as far as 
I remember. Official Ant version to build is 1.8.2, but later ones also work.

Please clean up your build directory (or checkout a new one).

Uwe

Am November 4, 2019 12:07:46 PM UTC schrieb Martin Gainty :
having used xsl parsing in other projects to create HTML I am attempting to
run the ant build script lucene/build.xml
  
  
  
  
  
  
  
  


but the build errors out on lucene\src\main\xml\ENTITY_TermQuery.xml
attempting to find the Unknown host www.bar.xyz in

so maybe my ant 1.10 is too old to parse the input xml?
i tried running xsl thru maven xml-maven-plugin





From: Uwe Schindler 
Sent: Monday, November 4, 2019 6:06 AM
To: dev@lucene.apache.org 
Subject: RE: dtd anomaly


Hi,



I am wondering what you are doing. If you run “ant documentation” from Solr’s 
or Lucene’s root folder, it should not even read those files – they are only 
relevant for the XML queryparser. Could it be that you have accidentally copied 
into some other folder where they are caught by some filename pattern? IMHO, 
the files should only be in XML query parser, but not in Lucene’s core.



The files are there to test correct handling of external entities so they 
should be in some test folder.



What are you exactly doing?



Uwe



-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

https://www.thetaphi.de

eMail: u...@thetaphi.de



From: Martin Gainty 
Sent: Monday, November 4, 2019 11:54 AM
To: dev@lucene.apache.org
Subject: dtd anomaly



here is a bug i cannot shake in when building lucene/site

inside lucene/src/main/xml/ENTITY_TermQuery.xml





http://www.bar.xyz/external;>

http://www.bar.xyz/param;>


using ant build.xml:
 



  

  

  

  

  

  

  





OR maven pom.xml

  

org.codehaus.mojo

   xml-maven-plugin

   1.0.1

   



 validate

initialize



 transform





   true

   false

   
${project.build.directory}/target

 

   

  src/main/xml

  C:/Maven-plugin/lucene-solr/lucene/site/xsl/index.xsl

  

   

 MyParam

 true

   

   

   

 

   

   

   

   



 net.sf.saxon

 Saxon-HE

 9.9.1-1



   

  



either build executing XSLT i get the same error:

[ERROR] Failed to execute goal 
org.codehaus.mojo:xml-maven-plugin:1.0.1:transform (validate) on project 
analysis: Failed to transform input file 
lucene/src/main/xml/ENTITY_TermQuery.xml: I/O error reported by XML parser 
processing file://lucene/src/main/xml/ENTITY_TermQuery.xml: 
www.bar.xyz:
Unknown host www.bar.xyz

]>

apparently www.bar.xyz host is supposed to be a placeholder
but for the life of me I cannot see where www.bar.zyz 
placeholder is replaced by a valid URL

(i havent used DTD in at least 10 years and i am way out of my element when 
trying to resolve)
any suggestions?

martin



--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

Re: SolrCloud is sick.

[Mark Miller]

There are 10,000 problems here.

So if you eventually land on one possible solution you agree on, we a
little closer.

There is no problem with the current design. Design's can always be
improved, sure. I've made this one fast. You won't believe me fast. The low
hanging fruit is astronomical, there is more fruit above that.

We never focused on performance. Or at least didn't. That's after we harden.

Except performance is the key to everything.

SolrCloud is not the only problem. The design of Solr, of SolrCloud, they
are fine. Change them, I don't care. Later. They are not a problem.

But Solr has as many problems as SolrCloud at this point. This just mater
a whole hell of lot less unless they are messing with SolrCloud. Standalone
is more of a brute.

We have 60 modules that are interconnected. We have a huge code base. That
is also fine.

We don't tend our garden. That's not fine. I've tended the garden before
without one - more than once before. It's a great damn garden. You guys
only get to see it grown over and full of weeds.

Anyway, no redesign, no library, no nothing like that gonna save this.

This is hardly concrete awareness of a problem here. The awareness to
figure out what actually are the problems and what must be done - that's
expensive shit these days if you ask me. I've been wrong lots tough.






On Mon, Nov 4, 2019 at 2:26 PM Jörn Franke  wrote:

> I guess this is also a bit normal with software that grows over the years.
> One could also say that one writes the current use cases and interesting
> future use cases for Solr in a document and designs from scratch new -
> taking only the good pieces out of the existing software.
> Of course there is a certain amount of time where you need to maintain
> both - but this will be also the case for a major rewrite.
>
> > Am 04.11.2019 um 20:58 schrieb Erick Erickson :
> >
> > If Curator would make that easier and we’re doing major surgery anyway….
> >
> > But yeah, a nifty, new, more modern tool isn’t going to magically help
> if the design is flawed.
> >
> > Or, if I’m putting my philosophical hat on, code doesn’t get gnarly
> intentionally. It gets gnarly because there are a bunch of problems to be
> solved and you don’t know what they are until you run into them. And it’s
> always a tension between fixing it enough to get by and fixing it by
> refactoring/redesign.
> >
> > But eventually “fixing it enough to get by” totters under it’s own
> weight and becomes increasingly fragile and you must take the hit and redo
> major portions of it. The questions now are:
> > 1> are we at that point?
> > 2> are we going to put the effort into rewriting some of the worst
> offenders?
> >
> >
> >
> >> On Nov 4, 2019, at 1:28 PM, Scott Blum  wrote:
> >>
> >> Figuring out a better overall algorithmic & data structure design
> that's an order of magnitude improvement seems far more important than
> swapping out libraries.  And I say this as a Curator fan and committer. ;)
> >>
> >> On Mon, Nov 4, 2019 at 11:44 AM Erick Erickson 
> wrote:
> >> Bram:
> >>
> >> Using Curator has been proposed before. It would require significant
> refactoring b/c of how deeply entwined raw ZK is in the code. That said, if
> we’re going to do major surgery it may be the right time to consider it.
> >>
> >> Erick
> >>
>  On Nov 4, 2019, at 9:24 AM, Bram Van Dam 
> wrote:
> >>>
>  SolrCloud is sick right now. The way low level Zookeeper is handeled
> >>>
> >>> On an unrelated project, I've stopped using "raw" ZK client access and
> >>> have switched to Curator. The API is a fair bit easier to work with,
> and
> >>> it results in less ugly code. I realize that this won't go very far in
> >>> resolving more fundamental issues, but it might be something that can
> >>> help improve the shape of the code.
> >>>
> >>> - Bram
> >>>
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> >>> For additional commands, e-mail: dev-h...@lucene.apache.org
> >>>
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> >> For additional commands, e-mail: dev-h...@lucene.apache.org
> >>
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

-- 
- Mark

http://about.me/markrmiller