[
https://issues.apache.org/jira/browse/SOLR-14788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17235852#comment-17235852
]
Mark Robert Miller commented on SOLR-14788:
-------------------------------------------
My initial implementation only really focused on a single collection - even
that was far, far from completed. Now I was not involved in Overseer
implementation, but it was not introduced to batch updates, to one state.json
or several - at least nothing like what it was doing. If that was even the
drive for it (it wasn't from my memory and knowledge) it would have been very
silly to try and handle our absurd state.json update load via an Overseer node
before making all the other nodes try and behave even remotely sane.
That Overseer did nothing by batching between collections - that was like
adding a bucket of water to a fire truck that is out of water. That is mostly
what has happened unfortunately - bandaids and workarounds. I never implemented
the SolrCloud Yonik and I worked out. I started it. We had the design. I put a
foot in that direction. Since then, things have mostly gone down that foot hole
instead of forward. Likey, as was the case for me, for many, it was not their
job to finish implementing SolrCloud, it was a huge task, few understood what
the actual design was, and you could do quite well riding on what was there for
little effort vs a lot of effort and who knows where you end up.
The Overseer as implemented was not in line with the design. This is an event
driven design. A light weight, low cost, simple design. Building it on an
existing and non Cloud oriented design made it very difficult to decipher what
the plan actually was or even how/if you could get there on these building
blocks while keeping them stable and active and non cloud mode, etc.
So when I talk about the benefits the Overseer type nodes can bring, they
hardly apply to master. It's a common problem I've run into. I'll talk about
how slow something is, or how much better things can be if do X, and someone
might take a little look and come back with, meh, didn't seem like what you
were saying to me. And often, there are so many layers that you can't see much
benefit or any when you play around with some isolated change in the current
world. 10 other things will eat you first.
Anyway, the system started by distributing updates without the help of a
central server :) The Overseer was not created to deal with clusterstate.json,
because we did not have state.json, that would be crazy :) It literally serves
no practical purpose at this point, other than a huge amount of problems and
slowness and bad behavior.
Now, I'm excited for any competition on what direction to go here. Don't take
any of this negatively. If your CAS system can run the gauntlet, I'll
congratulate you and be thankful. But your responses and the details in the
remove the Overseer issue seem (as is common enough) overly caught up in the
current nonsensical SolrCloud world. I wish you the best of luck making this
system and what it does and supports hum without a central server(s). It was
what I tried to keep in the design at the start. But it loses when you run the
mind simulations and ignore the current SolrCloud baggage and it almost
certainly loses when you implement it. You will have to shoot for the moon
though, not the current Overseer implementation, because my challenger is
almost to the ring and is in a different weight class / league / world than
what you have evaluated in 8x/master.
> Solr: The Next Big Thing
> ------------------------
>
> Key: SOLR-14788
> URL: https://issues.apache.org/jira/browse/SOLR-14788
> Project: Solr
> Issue Type: Task
> Reporter: Mark Robert Miller
> Assignee: Mark Robert Miller
> Priority: Critical
>
> h3.
> [!https://www.unicode.org/consortium/aacimg/1F46E.png!|https://www.unicode.org/consortium/adopted-characters.html#b1F46E]{color:#00875a}*The
> Policeman is on duty!*{color}
> {quote}_{color:#de350b}*When The Policeman is on duty, sit back, relax, and
> have some fun. Try to make some progress. Don't stress too much about the
> impact of your changes or maintaining stability and performance and
> correctness so much. Until the end of phase 1, I've got your back. I have a
> variety of tools and contraptions I have been building over the years and I
> will continue training them on this branch. I will review your changes and
> peer out across the land and course correct where needed. As Mike D will be
> thinking, "Sounds like a bottleneck Mark." And indeed it will be to some
> extent. Which is why once stage one is completed, I will flip The Policeman
> to off duty. When off duty, I'm always* {color:#de350b}*occasionally*{color}
> *down for some vigilante justice, but I won't be walking the beat, all that
> stuff about sit back and relax goes out the window.*{color}_
> {quote}
>
> I have stolen this title from Ishan or Noble and Ishan.
> This issue is meant to capture the work of a small team that is forming to
> push Solr and SolrCloud to the next phase.
> I have kicked off the work with an effort to create a very fast and solid
> base. That work is not 100% done, but it's ready to join the fight.
> Tim Potter has started giving me a tremendous hand in finishing up. Ishan and
> Noble have already contributed support and testing and have plans for
> additional work to shore up some of our current shortcomings.
> Others have expressed an interest in helping and hopefully they will pop up
> here as well.
> Let's organize and discuss our efforts here and in various sub issues.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
