[
https://issues.apache.org/jira/browse/SOLR-14788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225833#comment-17225833
]
Mark Robert Miller commented on SOLR-14788:
-------------------------------------------
Okay, things are really rolling. I hate to make it seem like things will never
be done as I solve one more issue, but hardening our worst tests forced me to
take one more look at the Overseer.
The last time I mad a Mark Miller Solr branch, after working with our
distributed queue implementation, I settled on the fact that it was terrible
for our use case and the implementation was lacking regardless. At that point I
tossed the Overseer and started on the SolrSeer, doing things more how I’d
approach them. This time, trying to be conservative, I settled on trying to
make the existing queue and logic as good as it can be. Progress not
perfection, we can deal with the Overseer more fully later.
However, as I started in on the unignores for some of our most flakey and
problematic and sometimes esoteric tests, it became clear things were
fundamentally flawed in a way that can’t be punted.
The way that we did things incrementally put us in an absurd place in terms of
collection/config admin and cluster state structure and Replica/Shard ACTIVE
states.
The fact that we have two overseer threads and queues, each with different
levels of concurrency and blocking, is very problematic because the commands
from the queue are dependent on each other.
When you add in the fact that we essentially have a split brain in terms of who
really owns the clusterstate, and still attempted mix support of predefined
cores or collection API cores, and the problems just flow on and on from there.
So as I looked at making that whole bag of ugliness reasonable, the solution is
very close to the approach I was going for with the SolrSeer.
When you put it on top of the work I already did with the ZKStateWriter, it’s a
game changer that I won’t even try to do credit, it’s something that will
happily speak for itself.
With great caution, I hazard that that is my last lack of sleep, binge effort.
I still have to call back my mom from a missed call last week - it’s been a
constant question in my household- where the hell is Mark, I don’t think he
lives here :) I’m ready for it to end as much as anyone.
So I wrap this up and hit the rest of those tests ASAP, non binge style, and
then it’s catch up to master. It doesn’t ever quite seem it, but it’s coming.
It’s coming soon.
> Solr: The Next Big Thing
> ------------------------
>
> Key: SOLR-14788
> URL: https://issues.apache.org/jira/browse/SOLR-14788
> Project: Solr
> Issue Type: Task
> Reporter: Mark Robert Miller
> Assignee: Mark Robert Miller
> Priority: Critical
>
> h3.
> [!https://www.unicode.org/consortium/aacimg/1F46E.png!|https://www.unicode.org/consortium/adopted-characters.html#b1F46E]{color:#00875a}*The
> Policeman is on duty!*{color}
> {quote}_{color:#de350b}*When The Policeman is on duty, sit back, relax, and
> have some fun. Try to make some progress. Don't stress too much about the
> impact of your changes or maintaining stability and performance and
> correctness so much. Until the end of phase 1, I've got your back. I have a
> variety of tools and contraptions I have been building over the years and I
> will continue training them on this branch. I will review your changes and
> peer out across the land and course correct where needed. As Mike D will be
> thinking, "Sounds like a bottleneck Mark." And indeed it will be to some
> extent. Which is why once stage one is completed, I will flip The Policeman
> to off duty. When off duty, I'm always* {color:#de350b}*occasionally*{color}
> *down for some vigilante justice, but I won't be walking the beat, all that
> stuff about sit back and relax goes out the window.*{color}_
> {quote}
>
> I have stolen this title from Ishan or Noble and Ishan.
> This issue is meant to capture the work of a small team that is forming to
> push Solr and SolrCloud to the next phase.
> I have kicked off the work with an effort to create a very fast and solid
> base. That work is not 100% done, but it's ready to join the fight.
> Tim Potter has started giving me a tremendous hand in finishing up. Ishan and
> Noble have already contributed support and testing and have plans for
> additional work to shore up some of our current shortcomings.
> Others have expressed an interest in helping and hopefully they will pop up
> here as well.
> Let's organize and discuss our efforts here and in various sub issues.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
