I've synced back all changes to confluence. Both represent the same document as of now. Please review and suggest if there are any outstanding concerns. Thanks for all the feedback.
I wish to proceed with the implementation based on lazy consensus again. On Tue, Nov 23, 2021 at 9:38 PM Mike Drob <md...@mdrob.com> wrote: > I should clarify, that my -1 was specifically about reaching lazy > consensus and not about that proposal itself. > > On Tue, Nov 23, 2021 at 9:42 AM Ishan Chattopadhyaya < > ichattopadhy...@gmail.com> wrote: > >> > -1, I would like to see a proposal on list where it will be permanently >> available >> >> It will be available in the confluence document (as it currently is). The >> google docs is just for collecting temporary feedback, thanks to easy >> inline commenting capability. We promised to consolidate feedback in the >> gdoc and sync back to the SIP document in confluence. This is reasonable >> enough expectations, and shouldn't be a worry enough for a -1. >> >> As of right now, while trying to address some concerns, some edits have >> been made in gdocs by Noble that we'll sync back into confluence; so that's >> the only divergence. >> >> On Tue, Nov 23, 2021 at 9:09 PM Ishan Chattopadhyaya < >> ichattopadhy...@gmail.com> wrote: >> >>> > -1, I would like to see a proposal on list where it will be >>> permanently available >>> > in the archives instead of being directed to a Google doc which can be >>> edited at any point in time. >>> > This has been an incredibly difficult proposal to keep track of. >>> >>> This entire SIP process has been incredibly difficult to coordinate. The >>> idea of doing this on a mail thread was a terrible one. There's no archive >>> available publicly for this list that doesn't break the threading >>> functionality (I guess someone among us is using some funky mail client >>> that's messing with Pony Mail). JIRA for discussion would've been much >>> better. I suppose JIRA used to have threaded discussions once upon a time, >>> that would've been very handy here. >>> >>> On Tue, Nov 23, 2021 at 9:07 PM Ishan Chattopadhyaya < >>> ichattopadhy...@gmail.com> wrote: >>> >>>> Okay, sure, I'll rescind my assertion about having reached a lazy >>>> consensus. I'll work through all the recent comments and feedback. >>>> I think at the last moment, there was an edit by Noble to the Google >>>> Doc that hasn't made it into the SIP document, and I'll review that change, >>>> the recent comments and get back here tomorrow. Other than that last moment >>>> change, the SIP document is up to date. >>>> >>>> On Tue, Nov 23, 2021 at 8:24 PM Gus Heck <gus.h...@gmail.com> wrote: >>>> >>>>> Also, I don't know that it's defined anywhere, but I feel that >>>>> declaring lazy consensus on a SIP in less than a week is rushing things. >>>>> Some folks won't have the flexibility to address things daily and I've >>>>> been >>>>> uncomfortably pulled away from paid work trying to keep up with this as >>>>> is. >>>>> Lazy consensus should include at least one weekend (IMHO) >>>>> >>>>> On Tue, Nov 23, 2021 at 9:49 AM Gus Heck <gus.h...@gmail.com> wrote: >>>>> >>>>>> IMHO two things must happen before lazy consensus wait period can >>>>>> begin: >>>>>> 1) The wiki must be updated to reflect the results of discussion in >>>>>> the google doc. >>>>>> 2) The fact that the wiki has been updated, and that you think the >>>>>> discussion is at an end must be posted here, preferably with a summary on >>>>>> the list. >>>>>> >>>>>> If it didn't happen on the list it didn't happen. >>>>>> >>>>>> Also the google doc has not been notifying me of changes/responses. >>>>>> So a note here when you've responded and some time to respond before the >>>>>> above steps are taken would be appreciated. >>>>>> >>>>>> -Gus >>>>>> >>>>>> On Tue, Nov 23, 2021 at 9:40 AM Mike Drob <md...@mdrob.com> wrote: >>>>>> >>>>>>> -1, I would like to see a proposal on list where it will be >>>>>>> permanently available in the archives instead of being directed to a >>>>>>> Google >>>>>>> doc which can be edited at any point in time. >>>>>>> >>>>>>> This has been an incredibly difficult proposal to keep track of. >>>>>>> >>>>>>> Mike >>>>>>> >>>>>>> On Tue, Nov 23, 2021 at 6:09 AM Ishan Chattopadhyaya < >>>>>>> ichattopadhy...@gmail.com> wrote: >>>>>>> >>>>>>>> This proposal has passed with lazy consensus. We can proceed to the >>>>>>>> implementation phase. >>>>>>>> Thanks to everyone for feedback, esp. Gus for the patience. >>>>>>>> >>>>>>>> On Sun, Nov 21, 2021 at 2:24 PM Gus Heck <gus.h...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks for adding that :) and Thanks to Nobel for translating my >>>>>>>>> ravings :). Sorry about the acronym. Amusingly I had to look up gish >>>>>>>>> gallop >>>>>>>>> to understand what you meant :) I can assure you that I will never >>>>>>>>> gish >>>>>>>>> gallop you or anyone on the list, and I find such techniques >>>>>>>>> repugnant. If >>>>>>>>> I'm not making sense, I'm likely covering ground too fast, skipping >>>>>>>>> things >>>>>>>>> that are in my head but not yours, expressing too many tangents (when >>>>>>>>> I >>>>>>>>> start nesting parentheses this is (usually) a sign I'm wandering too >>>>>>>>> much), >>>>>>>>> or I am simply mistaken about something. Always feel free to ask me to >>>>>>>>> explain! >>>>>>>>> >>>>>>>>> -Gus >>>>>>>>> >>>>>>>>> On Sun, Nov 21, 2021 at 12:17 AM Ishan Chattopadhyaya < >>>>>>>>> ichattopadhy...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Added both your suggestions to Google Docs for inline commenting >>>>>>>>>> and the SIP document at confluence. Thanks for the feedback. >>>>>>>>>> >>>>>>>>>> On Sun, Nov 21, 2021 at 10:28 AM Ishan Chattopadhyaya < >>>>>>>>>> ichattopadhy...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> I'll add the lifecycle details to the document as well. >>>>>>>>>>> >>>>>>>>>>> On Sun, 21 Nov, 2021, 9:08 am Ishan Chattopadhyaya, < >>>>>>>>>>> ichattopadhy...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Gus, >>>>>>>>>>>> Thanks for bringing it up again. Initially, I wasn't clear what >>>>>>>>>>>> you meant and mistook them for gish gallop. >>>>>>>>>>>> >>>>>>>>>>>> These were the things I am/was confused about (Noble explained >>>>>>>>>>>> it to me what you mean): >>>>>>>>>>>> - adherence to DRY via zk >>>>>>>>>>>> - runtime config info should come from zk >>>>>>>>>>>> - add dynamic features >>>>>>>>>>>> - same interaction patterns for consulting config values. >>>>>>>>>>>> >>>>>>>>>>>> Now, after Noble's help, I think I understand your motivations. >>>>>>>>>>>> - DRY = Don't Repeat Yourself ;-) >>>>>>>>>>>> - You're suggesting that we have a standard way to have role >>>>>>>>>>>> specific configurations (in ZK), so that a new role added later >>>>>>>>>>>> can access >>>>>>>>>>>> the configurations in a standard way >>>>>>>>>>>> >>>>>>>>>>>> This is a very good suggestion, and makes complete sense now. >>>>>>>>>>>> This was definitely a missing piece! >>>>>>>>>>>> >>>>>>>>>>>> Here's a proposal to address that (adding it to Google Docs: >>>>>>>>>>>> https://docs.google.com/document/d/1hijvM1WX9u2TOUdLEkFYVCofLeJlv2MRZqe-ncobJVw/edit >>>>>>>>>>>> ): >>>>>>>>>>>> - Per Role ZNode >>>>>>>>>>>> - Every role's ZNode to hold configuration data (this is the >>>>>>>>>>>> key addition that you're asking for, IIUC) >>>>>>>>>>>> - Children of role ZNodes to be list of ephemeral solr nodes >>>>>>>>>>>> >>>>>>>>>>>> We can iterate over this on the Google Docs (internal >>>>>>>>>>>> representation ZK section) with inline commenting if we need to. >>>>>>>>>>>> >>>>>>>>>>>> Please let us know how that sounds. >>>>>>>>>>>> Regards, >>>>>>>>>>>> Ishan >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Sun, Nov 21, 2021 at 4:49 AM Gus Heck <gus.h...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> It is difficult to track everything in this thread for sure. >>>>>>>>>>>>> Specifically, I don't feel my email of Wed, Nov 17, 7:43 PM (EST) >>>>>>>>>>>>> has been >>>>>>>>>>>>> addressed or responded to except for one difference of opinion >>>>>>>>>>>>> with Jan on >>>>>>>>>>>>> whether or not we should prohibit the notion of any role ever >>>>>>>>>>>>> changing at >>>>>>>>>>>>> runtime. >>>>>>>>>>>>> >>>>>>>>>>>>> Note that I made a specific proposal for an addition to the >>>>>>>>>>>>> sip, regarding start up lifecycle and adherence to DRY via zk, in >>>>>>>>>>>>> the >>>>>>>>>>>>> (probably not clearly expressed) hope that we can move to a >>>>>>>>>>>>> general overall >>>>>>>>>>>>> application principle that at runtime config info should come >>>>>>>>>>>>> from zk. If >>>>>>>>>>>>> we follow such a principle, we will always be in a position to >>>>>>>>>>>>> add dynamic >>>>>>>>>>>>> features without significant rework, and all code could hopefully >>>>>>>>>>>>> reuse the >>>>>>>>>>>>> same interaction patterns for consulting config values. >>>>>>>>>>>>> >>>>>>>>>>>>> Neither item is addressed in the SIP currently. >>>>>>>>>>>>> >>>>>>>>>>>>> Also I had put that out there as a single item so it could be >>>>>>>>>>>>> focused on (simplifying the discussion I hope), and depending on >>>>>>>>>>>>> how >>>>>>>>>>>>> discussion proceeds, I may have other follow on items. >>>>>>>>>>>>> >>>>>>>>>>>>> -Gus >>>>>>>>>>>>> >>>>>>>>>>>>> On Sat, Nov 20, 2021 at 8:45 AM Ishan Chattopadhyaya < >>>>>>>>>>>>> ichattopadhy...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, 19 Nov, 2021, 7:03 pm Ilan Ginzburg, < >>>>>>>>>>>>>> ilans...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Is the request here for everybody to express again the >>>>>>>>>>>>>>> concerns >>>>>>>>>>>>>>> already expressed in this email thread and not addressed? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think the expectation now (after we've expressed our >>>>>>>>>>>>>> intention to reach a lazy consensus) is that if someone wants to >>>>>>>>>>>>>> block this >>>>>>>>>>>>>> SIP from adoption, then to put forward the objections. Sort of >>>>>>>>>>>>>> like a veto. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I suggest instead the authors review the thread, match >>>>>>>>>>>>>>> expressed >>>>>>>>>>>>>>> concerns with how the concern was addressed (or not >>>>>>>>>>>>>>> addressed) and >>>>>>>>>>>>>>> provide an exhaustive list. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I've summarized most of the discussion in the SIP document. >>>>>>>>>>>>>> If you feel I could do a better job with it, please help me with >>>>>>>>>>>>>> areas of >>>>>>>>>>>>>> improvement. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This proposal in its current form (data and overseer roles) >>>>>>>>>>>>>>> doesn't >>>>>>>>>>>>>>> offer much that can't be reasonably achieved by other means. >>>>>>>>>>>>>>> I'd find >>>>>>>>>>>>>>> much more value in making sure what is done now is a solid >>>>>>>>>>>>>>> foundation >>>>>>>>>>>>>>> for the future. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Fair enough, I understand your perspective. Thanks for your >>>>>>>>>>>>>> feedback. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Ilan >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Nov 18, 2021 at 11:24 PM Noble Paul < >>>>>>>>>>>>>>> noble.p...@gmail.com> wrote: >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > After so many back and forth mails, I just can't say who >>>>>>>>>>>>>>> has an outstanding concern and if they are already addressed or >>>>>>>>>>>>>>> not. I >>>>>>>>>>>>>>> think the Google doc would help us get clarity on that. Please >>>>>>>>>>>>>>> take a >>>>>>>>>>>>>>> moment to give your inputs >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > On Fri, Nov 19, 2021, 9:18 AM Ishan Chattopadhyaya < >>>>>>>>>>>>>>> ichattopadhy...@gmail.com> wrote: >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> Apologies, the vote hasn't passed formally and I was >>>>>>>>>>>>>>> under some confusion on the process. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> I'd like to proceed with a lazy consensus and proceed to >>>>>>>>>>>>>>> the implementation phase now. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> However, I would appreciate it if someone wants to bring >>>>>>>>>>>>>>> out any outstanding concerns about the SIP document. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> To facilitate in-line comments, here's a temporary Google >>>>>>>>>>>>>>> Docs version of this document. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> https://docs.google.com/document/d/1hijvM1WX9u2TOUdLEkFYVCofLeJlv2MRZqe-ncobJVw/edit?usp=sharing >>>>>>>>>>>>>>> >> (I shall copy changes back to confluence eventually) >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> Thanks and apologies again regarding the confusion with >>>>>>>>>>>>>>> the voting, >>>>>>>>>>>>>>> >> Regards, >>>>>>>>>>>>>>> >> Ishan >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> On Thu, Nov 18, 2021 at 9:50 PM Ishan Chattopadhyaya < >>>>>>>>>>>>>>> ichattopadhy...@gmail.com> wrote: >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> The SIP passed the voting phase. Thanks for all for the >>>>>>>>>>>>>>> feedback and insights. >>>>>>>>>>>>>>> >>> Looking forward to your collaboration and reviews as we >>>>>>>>>>>>>>> implement this. >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> On Thu, Nov 18, 2021 at 9:42 PM Ishan Chattopadhyaya < >>>>>>>>>>>>>>> ichattopadhy...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>>> > It's fine if we don't provide any ability for runtime >>>>>>>>>>>>>>> modification of roles at this time but I'm leery of precluding >>>>>>>>>>>>>>> it in the >>>>>>>>>>>>>>> future. >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>>> In future, the necessity for such a facility can >>>>>>>>>>>>>>> dictate our course of action. We cannot lay down rules cast in >>>>>>>>>>>>>>> stone for >>>>>>>>>>>>>>> functionality that we can't foresee yet. >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>>> On Thu, Nov 18, 2021 at 9:40 PM Ishan Chattopadhyaya < >>>>>>>>>>>>>>> ichattopadhy...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>> >>>>> Thanks Jan, I added both those points to the SIP >>>>>>>>>>>>>>> document in the Notes section. >>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>> >>>>> On Thu, Nov 18, 2021 at 7:18 PM Jan Høydahl < >>>>>>>>>>>>>>> jan....@cominvent.com> wrote: >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> >>>>>> 18. nov. 2021 kl. 01:43 skrev Gus Heck < >>>>>>>>>>>>>>> gus.h...@gmail.com>: >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> >>>>>> 2) Roles will not be checked by loading config from >>>>>>>>>>>>>>> disk or caching disk config in memory. (zk ONLY source of truth) >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> >>>>>> It sounds a bit backward for a local node to first >>>>>>>>>>>>>>> parse solr.node.roles, determine its local set of roles, then >>>>>>>>>>>>>>> publish them >>>>>>>>>>>>>>> to Zookeeper, and then read back its own roles from ZK. >>>>>>>>>>>>>>> >>>>>> Code that only needs to determine "Do I have the XXX >>>>>>>>>>>>>>> role?" or find out "What roles do I have" should be able to >>>>>>>>>>>>>>> fetch the >>>>>>>>>>>>>>> (static) roles from some roles utility class without consulting >>>>>>>>>>>>>>> ZK. >>>>>>>>>>>>>>> >>>>>> Code that needs to check what nodes have a certain >>>>>>>>>>>>>>> role (such as placement) would obviously need ot consult ZK. >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> >>>>>> Perhaps the SIP should also state some Non-goals or >>>>>>>>>>>>>>> assertions such as >>>>>>>>>>>>>>> >>>>>> * Roles are static and immutable (also in zk) for the >>>>>>>>>>>>>>> entire life cycle of a node >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> >>>>>> I also think we should state that the bar for adding >>>>>>>>>>>>>>> new roles should be high so it is not abused as any other tag >>>>>>>>>>>>>>> or label for >>>>>>>>>>>>>>> any tiny feature. It should be reserved for functionality that >>>>>>>>>>>>>>> may benefit >>>>>>>>>>>>>>> from a dedicated set of nodes. That may be clear already, but >>>>>>>>>>>>>>> you never >>>>>>>>>>>>>>> know... >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> >>>>>> Jan >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >>>>>>>>>>>>>>> For additional commands, e-mail: dev-h...@solr.apache.org >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> http://www.needhamsoftware.com (work) >>>>>>>>>>>>> http://www.the111shift.com (play) >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> http://www.needhamsoftware.com (work) >>>>>>>>> http://www.the111shift.com (play) >>>>>>>>> >>>>>>>> >>>>>> >>>>>> -- >>>>>> http://www.needhamsoftware.com (work) >>>>>> http://www.the111shift.com (play) >>>>>> >>>>> >>>>> >>>>> -- >>>>> http://www.needhamsoftware.com (work) >>>>> http://www.the111shift.com (play) >>>>> >>>>
