WHO OVERSEES THE OVERSEER???? On Tue, Nov 5, 2019 at 5:16 PM Mark Miller <markrmil...@gmail.com> wrote:
> bq. SolrCloud is a ballerina. Doesn't look it, cause we dont take care > of it. > > And this is why I'm so devastated by the Overseer. I don't blame anyone > person. Where was the manual, where was my intervention. I whispered > Overseer and cut one more thing off my list of responsibilities. > > But the overseer is supposed to be so light weight and easy breezy. Giving > up leader shop at the most signs of trouble, keeping communication small > and tight with tiny json distrib queue pub/sub updates. Little about stat > change, hardly needed. Hardly ever talking to Zookeeper. > > Our whole system is not moved hard against this, but nothing so much as > the Overseer. It has very scary, very tricky, custom ZK code. It has major > communication with ZK. It has little to weak ability to properly throttle > itself or deal with things intelligently. It's almost a brute force tactic. > And it clings to being Overseer like a moth to flame. It's designed to be > on a dedicated hardwar and then mostly to not make any reasonable use of > that hardware. > > I blame me more than anyone for that. I am mad at me. It's just an > absolute brain bash with a sledge hammer to the system. And i never > communicated the system very well. I was overloaded. > > On Tue, Nov 5, 2019 at 11:01 AM Mark Miller <markrmil...@gmail.com> wrote: > >> And now we are to meat of it. Fill in here: >> https://issues.apache.org/jira/browse/SOLR-13888 >> >> We can play in a better world, we can have fun, but some of you are going >> to have to adjust your ways. In the most convenient way possible. You are >> all great people, I don't want to cause you annoyance, but there are >> certain requirements to building an aircraft, and there certain >> requirements to building this. >> >> On Tue, Nov 5, 2019 at 10:44 AM Mark Miller <markrmil...@gmail.com> >> wrote: >> >>> If you had any idea how much suffering just that has caused. Not just >>> users, but us. >>> >>> Mark >>> >>> On Tue, Nov 5, 2019 at 10:38 AM Mark Miller <markrmil...@gmail.com> >>> wrote: >>> >>>> It’s like 6-7 years since I quickly added a shitty collections API in >>>> my free time because we desperately needed SOMETHING. I don’t know if I >>>> tried to make it wait for proper state or what , it was a stub to try get >>>> things moving. That call, to this day, along with all our other checks, >>>> until some tests ones recently, is garbage. >>>> >>>> If I downloaded a database, and a lot the time, after the create a >>>> database call returned, my database was not ready, I’d saw wow. Terrible >>>> bug got through. If it was a persistent issue for over half a decade? My >>>> god. >>>> >>>> Look I just spent that half decade upgrading from Solr 4 to whatever. I >>>> was mostly out of the loop. But this is crazy, me in there too. >>>> >>>> Mark >>>> >>>> On Tue, Nov 5, 2019 at 10:05 AM Mark Miller <markrmil...@gmail.com> >>>> wrote: >>>> >>>>> I'll tell you what guys, development right now sucks. I don't enjoy. >>>>> >>>>> But when I start to put things in shape? I get this smile, and I start >>>>> going with the feeling of I don't need you guys, I don't users, I don't >>>>> need a job, cause just this is figgen nice. >>>>> >>>>> On Tue, Nov 5, 2019 at 9:59 AM Mark Miller <markrmil...@gmail.com> >>>>> wrote: >>>>> >>>>>> I suppose I should toss one more out. >>>>>> >>>>>> Hell yes, we will be using curator. >>>>>> >>>>>> It's insane for any group larger than 2-3 to directly use ZooKeeper. >>>>>> Even for that group, you want some damn good reasons to not use curator. >>>>>> We >>>>>> can start using more assembly too (joke Yonik). >>>>>> >>>>>> Curator was an option initially. Then it was yet another project >>>>>> hosted by Netflix. Now it is essential. >>>>>> >>>>>> >>>>>> - Mark >>>>>> >>>>>> On Tue, Nov 5, 2019 at 9:41 AM Mark Miller <markrmil...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> And look, we started pretty deep in the hole. Solr started with tons >>>>>>> of bug or limitations that hardly mattered to it and hit SolrCloud in >>>>>>> the >>>>>>> eye like a train. And we were not setup to deal with that. >>>>>>> >>>>>>> We never had a nice garden for SolrCloud. We started in a mess, >>>>>>> thinking, eventually we clear the overgrowth, and we are all good. And >>>>>>> then >>>>>>> we started building our house and that garden went wild with a life of >>>>>>> it's >>>>>>> own. >>>>>>> >>>>>>> And our development practices, amazingly above many many many groups >>>>>>> and standards out there, is woefully inaccurate for what we are doing. >>>>>>> >>>>>>> "Test pass, I'm not sure about all this but I'm going to commit" >>>>>>> (Tests never pass, must be a lie anyway) >>>>>>> "Leaving on vacation, going to fire this in" >>>>>>> "No one has looked at this huge thing, it's been a while, going to >>>>>>> commit" >>>>>>> *commit* >>>>>>> >>>>>>> And comments to that affect pretty much wrap up our careful and >>>>>>> thoughtful attitude. >>>>>>> >>>>>>> And then of course we come and clean up after, careful gardeners >>>>>>> that we are ... no, we don't. We are not setup to be gardeners, we are >>>>>>> not >>>>>>> trying, even if we do, I only like grass and screw the other plants. >>>>>>> >>>>>>> Without SolrCloud, Solr wold be in trouble as well. Brute that it >>>>>>> is, it could go a few more rounds. SolrCloud is a ballerina. Doesn't >>>>>>> look >>>>>>> it, cause we dont take care of it. But it is, and it cannot take the >>>>>>> beating that the brute does. >>>>>>> >>>>>>> - Mark >>>>>>> >>>>>>> On Tue, Nov 5, 2019 at 5:19 AM Mark Miller <markrmil...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Basically I can fix 99% of this without you guys - with simple care >>>>>>>> and effort and time that non of you are likely in the circumstances of >>>>>>>> being able to duplicate.. Been there done that, made it 100x-1000x >>>>>>>> faster >>>>>>>> to boot and added all kinds of fun. >>>>>>>> >>>>>>>> But I can't build the rest of Solr. I don't care about facets. So >>>>>>>> let's meet half way. >>>>>>>> >>>>>>>> On Tue, Nov 5, 2019 at 5:14 AM Mark Miller <markrmil...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> There are 10,000 problems here. >>>>>>>>> >>>>>>>>> So if you eventually land on one possible solution you agree on, >>>>>>>>> we a little closer. >>>>>>>>> >>>>>>>>> There is no problem with the current design. Design's can always >>>>>>>>> be improved, sure. I've made this one fast. You won't believe me >>>>>>>>> fast. The >>>>>>>>> low hanging fruit is astronomical, there is more fruit above that. >>>>>>>>> >>>>>>>>> We never focused on performance. Or at least didn't. That's after >>>>>>>>> we harden. >>>>>>>>> >>>>>>>>> Except performance is the key to everything. >>>>>>>>> >>>>>>>>> SolrCloud is not the only problem. The design of Solr, of >>>>>>>>> SolrCloud, they are fine. Change them, I don't care. Later. They are >>>>>>>>> not a >>>>>>>>> problem. >>>>>>>>> >>>>>>>>> But Solr has as many problems as SolrCloud at this point. This >>>>>>>>> just mater a whole hell of lot less unless they are messing with >>>>>>>>> SolrCloud. Standalone is more of a brute. >>>>>>>>> >>>>>>>>> We have 60 modules that are interconnected. We have a huge code >>>>>>>>> base. That is also fine. >>>>>>>>> >>>>>>>>> We don't tend our garden. That's not fine. I've tended the garden >>>>>>>>> before without one - more than once before. It's a great damn garden. >>>>>>>>> You >>>>>>>>> guys only get to see it grown over and full of weeds. >>>>>>>>> >>>>>>>>> Anyway, no redesign, no library, no nothing like that gonna save >>>>>>>>> this. >>>>>>>>> >>>>>>>>> This is hardly concrete awareness of a problem here. The awareness >>>>>>>>> to figure out what actually are the problems and what must be done - >>>>>>>>> that's >>>>>>>>> expensive shit these days if you ask me. I've been wrong lots tough. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Nov 4, 2019 at 2:26 PM Jörn Franke <jornfra...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I guess this is also a bit normal with software that grows over >>>>>>>>>> the years. >>>>>>>>>> One could also say that one writes the current use cases and >>>>>>>>>> interesting future use cases for Solr in a document and designs from >>>>>>>>>> scratch new - taking only the good pieces out of the existing >>>>>>>>>> software. >>>>>>>>>> Of course there is a certain amount of time where you need to >>>>>>>>>> maintain both - but this will be also the case for a major rewrite. >>>>>>>>>> >>>>>>>>>> > Am 04.11.2019 um 20:58 schrieb Erick Erickson < >>>>>>>>>> erickerick...@gmail.com>: >>>>>>>>>> > >>>>>>>>>> > If Curator would make that easier and we’re doing major >>>>>>>>>> surgery anyway…. >>>>>>>>>> > >>>>>>>>>> > But yeah, a nifty, new, more modern tool isn’t going to >>>>>>>>>> magically help if the design is flawed. >>>>>>>>>> > >>>>>>>>>> > Or, if I’m putting my philosophical hat on, code doesn’t get >>>>>>>>>> gnarly intentionally. It gets gnarly because there are a bunch of >>>>>>>>>> problems >>>>>>>>>> to be solved and you don’t know what they are until you run into >>>>>>>>>> them. And >>>>>>>>>> it’s always a tension between fixing it enough to get by and fixing >>>>>>>>>> it by >>>>>>>>>> refactoring/redesign. >>>>>>>>>> > >>>>>>>>>> > But eventually “fixing it enough to get by” totters under it’s >>>>>>>>>> own weight and becomes increasingly fragile and you must take the >>>>>>>>>> hit and >>>>>>>>>> redo major portions of it. The questions now are: >>>>>>>>>> > 1> are we at that point? >>>>>>>>>> > 2> are we going to put the effort into rewriting some of the >>>>>>>>>> worst offenders? >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> >> On Nov 4, 2019, at 1:28 PM, Scott Blum <dragonsi...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >> >>>>>>>>>> >> Figuring out a better overall algorithmic & data structure >>>>>>>>>> design that's an order of magnitude improvement seems far more >>>>>>>>>> important >>>>>>>>>> than swapping out libraries. And I say this as a Curator fan and >>>>>>>>>> committer. ;) >>>>>>>>>> >> >>>>>>>>>> >> On Mon, Nov 4, 2019 at 11:44 AM Erick Erickson < >>>>>>>>>> erickerick...@gmail.com> wrote: >>>>>>>>>> >> Bram: >>>>>>>>>> >> >>>>>>>>>> >> Using Curator has been proposed before. It would require >>>>>>>>>> significant refactoring b/c of how deeply entwined raw ZK is in the >>>>>>>>>> code. >>>>>>>>>> That said, if we’re going to do major surgery it may be the right >>>>>>>>>> time to >>>>>>>>>> consider it. >>>>>>>>>> >> >>>>>>>>>> >> Erick >>>>>>>>>> >> >>>>>>>>>> >>>> On Nov 4, 2019, at 9:24 AM, Bram Van Dam < >>>>>>>>>> bram.van...@intix.eu> wrote: >>>>>>>>>> >>> >>>>>>>>>> >>>> SolrCloud is sick right now. The way low level Zookeeper is >>>>>>>>>> handeled >>>>>>>>>> >>> >>>>>>>>>> >>> On an unrelated project, I've stopped using "raw" ZK client >>>>>>>>>> access and >>>>>>>>>> >>> have switched to Curator. The API is a fair bit easier to >>>>>>>>>> work with, and >>>>>>>>>> >>> it results in less ugly code. I realize that this won't go >>>>>>>>>> very far in >>>>>>>>>> >>> resolving more fundamental issues, but it might be something >>>>>>>>>> that can >>>>>>>>>> >>> help improve the shape of the code. >>>>>>>>>> >>> >>>>>>>>>> >>> - Bram >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>>>>>>>>> >>> For additional commands, e-mail: dev-h...@lucene.apache.org >>>>>>>>>> >>> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>>>>>>>>> >> For additional commands, e-mail: dev-h...@lucene.apache.org >>>>>>>>>> >> >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>>>>>>>>> > For additional commands, e-mail: dev-h...@lucene.apache.org >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>>>>>>>>> For additional commands, e-mail: dev-h...@lucene.apache.org >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> - Mark >>>>>>>>> >>>>>>>>> http://about.me/markrmiller >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> - Mark >>>>>>>> >>>>>>>> http://about.me/markrmiller >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> - Mark >>>>>>> >>>>>>> http://about.me/markrmiller >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> - Mark >>>>>> >>>>>> http://about.me/markrmiller >>>>>> >>>>> >>>>> >>>>> -- >>>>> - Mark >>>>> >>>>> http://about.me/markrmiller >>>>> >>>> -- >>>> - Mark >>>> >>>> http://about.me/markrmiller >>>> >>> -- >>> - Mark >>> >>> http://about.me/markrmiller >>> >> >> >> -- >> - Mark >> >> http://about.me/markrmiller >> > > > -- > - Mark > > http://about.me/markrmiller >
