I've started a formal vote thread for the proposal here. Thanks all. On Thu, Nov 25, 2021 at 12:10 PM Ishan Chattopadhyaya < [email protected]> wrote:
> We have closed the Google Doc for further comments, and all changes have > been incorporated into the confluence document: > https://cwiki.apache.org/confluence/display/SOLR/SIP-15+Node+roles > Thanks for all the comments and feedback. > > On Thu, Nov 25, 2021 at 12:09 PM Ishan Chattopadhyaya < > [email protected]> wrote: > >> > Maybe one way to provide for the future is to place some requirements >> on roles, and designate that role-specific coordination information >> > be placed within the subtree for that role under >> /node-roles/<rolename>/ To facilitate this we could >> > 1. move the currently designed set of ephemeral nodes down one level >> into am "/available" node that reflects which nodes are currently up with >> the role (exactly what they do in the present design) >> >> We've introduced nesting in the ZK structure now. Ephemeral nodes for >> Solr node names will now be placed at >> /node-roles/<rolename>/nodes/<ephemeralnode>. >> >> >> >> >> >> >> On Thu, Nov 25, 2021 at 12:04 PM Ishan Chattopadhyaya < >> [email protected]> wrote: >> >>> >>> >>> On Thu, Nov 25, 2021 at 4:35 AM Gus Heck <[email protected]> wrote: >>> >>>> Things I think still need to be clarified to define "Role" explicitly. >>>> Primarily to guide future implementations. Things I can think of include: >>>> >>>> - What the bar for something to be a role vs being more appropriate >>>> as a property/tag type concept. Essentially what is and is not expected >>>> (at >>>> this time) to be a role. >>>> >>>> We don't wish to set a bar for anything at this point, since I can't >>> imagine all future innovation that can happen. All we can do is present >>> representative examples. Common sense will dictate what is useful to have >>> as a role, and clearly we can't set a bar for that. >>> >>> >>>> >>>> - What's the process for this? Do we want a SIP per role to ensure >>>> good discussion? Are there some simple things a role might do that are >>>> "normal" and don't require a SIP? Or maybe no sip generally? >>>> >>>> The community can decide on a case by case basis using established >>> community processes. I don't see the need to define a new process for this. >>> >>> >>>> >>>> - Where a role is defined (enum? class? which module? core? solrj?) >>>> and what motivates that choice (specifics such as methods/properties >>>> can be >>>> left to impl of course) >>>> >>>> This can be discussed during the implementation phase. This SIP is for >>> the general concept, public interface and other non-code concerns. I don't >>> want to start another 20 page thread here debating enum vs class and other >>> things that might tickle someone's fancy. Lets cross the bridge once we get >>> to it. >>> >>> >>>> >>>> - Naming conventions for roles... camel case? snake case? all caps? >>>> what characters are allowed, Possibly discourage any ideas with role >>>> names >>>> that embed config options such as overseer_priority_1 and any parsing of >>>> role names. >>>> >>>> >>> We can discuss that on a case by case basis when new roles are >>> introduced. I've already proposed that the sysprops for roles should be in >>> lowercase, which rules out camelCase. >>> >>> >>>> >>>> - >>>> >>>> If we answer those and bundle them with the 3 sections under the >>>> example api calls into a "Roles" section or "Definition of Roles" or >>>> something like that it seems good to me. >>>> >>>> Sure we could discuss any of this independently for each role in each >>>> sip but if we put guidance here we probably can skip that discussion for N >>>> out of M sips/tickets in the future (N being maximized if we choose well >>>> and write with clarity). Thus we'd be saving time in the long run and >>>> reduce the possibility that people propose something awkward and it slips >>>> through on lazy consensus. >>>> >>>> And of course it's Apache software so it can all be changed later by a >>>> vote or concensus. :) >>>> >>>> -Gus >>>> >>>> On Wed, Nov 24, 2021 at 3:21 PM Ishan Chattopadhyaya < >>>> [email protected]> wrote: >>>> >>>>> > Moreover I think it's a very role specific notion, that is not a good >>>>> > fit in the roles framework, so maybe should never be part of this >>>>> SIP. >>>>> >>>>> Regarding "capable" vs "currently providing", I agree with Ilan here ^ >>>>> >>>>> >>>>> >>>>> On Wed, Nov 24, 2021 at 10:38 PM Gus Heck <[email protected]> wrote: >>>>> >>>>>> You do make a good point that the details can be very role specific. >>>>>> Maybe one way to provide for the future is to place some requirements on >>>>>> roles, and designate that role-specific coordination information be >>>>>> placed >>>>>> within the subtree for that role under /node-roles/<rolename>/ To >>>>>> facilitate this we could >>>>>> >>>>>> 1. move the currently designed set of ephemeral nodes down one >>>>>> level into am "/available" node that reflects which nodes are >>>>>> currently up >>>>>> with the role (exactly what they do in the present design) >>>>>> 2. Roles must store role related info in the data for the role >>>>>> node, or in tree(s) that are peers to /available as they see fit. >>>>>> 3. Roles must have documentation detailing what they store where >>>>>> and how it's used. >>>>>> 4. Roles (beyond those implemented in this SIP) should be >>>>>> discussed in their own SIP (enforcing the non-trivial label concept >>>>>> we all >>>>>> seem to agree on) >>>>>> >>>>>> >>>>> I am not in favour of complicating the currently proposed design by >>>>> doing any of this now ^ If we need to, we can discuss this then. >>>>> >>>>> >>>>>> The bit that's not clear is where that documentation would live. It's >>>>>> not really material for the users, but more for the developers. >>>>>> >>>>>> On Wed, Nov 24, 2021 at 11:11 AM Ilan Ginzburg <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> I agree with Ishan that "currently providing" shouldn't be a current >>>>>>> concern for the SIP. >>>>>>> Moreover I think it's a very role specific notion, that is not a good >>>>>>> fit in the roles framework, so maybe should never be part of this >>>>>>> SIP. >>>>>>> >>>>>>> Even on the Overseer example. Suppose we changed the current design >>>>>>> to >>>>>>> have a "main" node being Overseer and a secondary to be somewhat >>>>>>> active standby in case the main one goes down (imagine some form of >>>>>>> replication between the two?). Both would have the "Overseer" role, >>>>>>> but what they provide would be different. Overseer specific code >>>>>>> should be able to know what is provided by which node, because a node >>>>>>> trying to use the Overseer service would have to talk to a specific >>>>>>> one of the two, not just one that currently provides. >>>>>>> >>>>>>> Same for data nodes. A data node can have replicas on it. But it may >>>>>>> not. If it doesn't, shall we say it has the capability but does not >>>>>>> provide? Of course not, nobody would suggest that. Because we know >>>>>>> that tracking the actual data is not a role level responsibility but >>>>>>> a >>>>>>> data (collection/shard/replica etc) tracking responsibility, done >>>>>>> using all the nice structures the code maintains in ZK and in caches >>>>>>> all over the place. >>>>>>> >>>>>>> Ilan >>>>>>> >>>>>>> On Wed, Nov 24, 2021 at 5:10 PM Gus Heck <[email protected]> wrote: >>>>>>> > >>>>>>> > So we are potentially talking past each other. Let me re-clarify >>>>>>> since you've interpreted "configuration" differently (I think). What I'm >>>>>>> referring to in ZK is the "reflection of configuration" not the actual >>>>>>> configuration. This is why I'm asking the question of how much info >>>>>>> should >>>>>>> be retained about a node after it goes down. Case in point zookeeper >>>>>>> embedded: The code might take different actions if it can see that a >>>>>>> node >>>>>>> that had a running zookeeper did exist (and thus may return) vs a >>>>>>> situation >>>>>>> where there's only 2 nodes playing zookeeper roles right now and no >>>>>>> evidence of a third ever having existed. >>>>>>> > >>>>>>> > Also you say you can't imagine anything beyond overseer... well I >>>>>>> could imagine that a cluster would want to say these 10 machines (but >>>>>>> not >>>>>>> the other 40) are allowed to have zookeeper, and I want the cluster to >>>>>>> to >>>>>>> have 3 node level of redundancy in zookeeper, and a new zookeeper >>>>>>> should be >>>>>>> created and the data replicated to it if a zk node is lost for more >>>>>>> than 30 >>>>>>> minutes. >>>>>>> > >>>>>>> > On Wed, Nov 24, 2021 at 10:38 AM Ishan Chattopadhyaya < >>>>>>> [email protected]> wrote: >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> On Wed, Nov 24, 2021 at 9:05 PM Ishan Chattopadhyaya < >>>>>>> [email protected]> wrote: >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> On Wed, Nov 24, 2021 at 8:19 PM Gus Heck <[email protected]> >>>>>>> wrote: >>>>>>> >>>> >>>>>>> >>>> Left some additional comments in the google doc, extracting the >>>>>>> key point for others here so they can know if they want to read the >>>>>>> google >>>>>>> doc commentary: >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> I'm addressing them here, inline. >>>>>>> >>> >>>>>>> >>>> >>>>>>> >>>> There is presently disagreement with my proposal that the sip >>>>>>> specify both how we track configuration (which I have been referring to >>>>>>> as >>>>>>> "capability") and runtime state ("providing"). I see the primary value >>>>>>> of >>>>>>> having a role concept as one of organizing the code and configuration >>>>>>> in a >>>>>>> way that is easy to understand, which is why I'm in favor of this SiP >>>>>>> overall. >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> I purposely don't want to complicate this design by allowing for >>>>>>> "capability" vs "currently providing" to be represented as first class >>>>>>> citizens. Besides the OVERSEER role (preferred overseer), I can't think >>>>>>> of >>>>>>> concrete scenarios where such a concept could be applicable, and hence I >>>>>>> don't want to get ahead of myself in trying to address how to solve >>>>>>> that. >>>>>>> Since the current proposal for dealing with roles is flexible and >>>>>>> generic >>>>>>> enough, I don't think adding such extensions would be a problem. >>>>>>> However, I >>>>>>> don't want to go down that rabbithole at this point in time. >>>>>>> >>> >>>>>>> >>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> If you don't want to tackle implementing the runtime state bit >>>>>>> I'm ok with that as long as we have a clear plan in the SIP of how this >>>>>>> type of information should be organized in the future. >>>>>>> >>>> The structure proposed in the SIP for information in zk seems >>>>>>> to map more clearly to runtime state than configuration, and seems like >>>>>>> it >>>>>>> would inhibit a natural representation of such. >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> The structure proposed allows for role specific configurations >>>>>>> (if needed). It doesn't provide for node specific configuration in ZK, >>>>>>> since all node properties should come from either solr.xml or sysprops. >>>>>>> At >>>>>>> present, I don't think we ever have node specific info configuration in >>>>>>> ZK >>>>>>> in any part of Solr, and I think we shouldn't go that route here. >>>>>>> >> >>>>>>> >> >>>>>>> >> Correction: "At present, I don't think we ever have node specific >>>>>>> configuration in ZK in any part of Solr, and I think we shouldn't go >>>>>>> that >>>>>>> route here." >>>>>>> >> >>>>>>> >>> >>>>>>> >>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> Finally the structure in the SIP for zk information doesn't >>>>>>> specify if some of the nodes are ephemeral, and reliable for a "current" >>>>>>> list or if they are persistent over time. I feel like we may not yet >>>>>>> have a >>>>>>> plan for how much information about nodes (n general) should be retained >>>>>>> while a node is down, which maybe needs to underlie this design. >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> All the node names under the roles znodes are ephemeral. Added >>>>>>> this to the document (confluence and gdoc both). >>>>>>> >>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> -Gus >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> On Tue, Nov 23, 2021 at 10:34 PM Ishan Chattopadhyaya < >>>>>>> [email protected]> wrote: >>>>>>> >>>>> >>>>>>> >>>>> I've synced back all changes to confluence. Both represent the >>>>>>> same document as of now. >>>>>>> >>>>> Please review and suggest if there are any outstanding >>>>>>> concerns. Thanks for all the feedback. >>>>>>> >>>>> >>>>>>> >>>>> I wish to proceed with the implementation based on lazy >>>>>>> consensus again. >>>>>>> >>>>> >>>>>>> >>>>> On Tue, Nov 23, 2021 at 9:38 PM Mike Drob <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>> >>>>>>> >>>>>> I should clarify, that my -1 was specifically about reaching >>>>>>> lazy consensus and not about that proposal itself. >>>>>>> >>>>>> >>>>>>> >>>>>> On Tue, Nov 23, 2021 at 9:42 AM Ishan Chattopadhyaya < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> > -1, I would like to see a proposal on list where it will >>>>>>> be permanently available >>>>>>> >>>>>>> >>>>>>> >>>>>>> It will be available in the confluence document (as it >>>>>>> currently is). The google docs is just for collecting temporary >>>>>>> feedback, >>>>>>> thanks to easy inline commenting capability. We promised to consolidate >>>>>>> feedback in the gdoc and sync back to the SIP document in confluence. >>>>>>> This >>>>>>> is reasonable enough expectations, and shouldn't be a worry enough for >>>>>>> a -1. >>>>>>> >>>>>>> >>>>>>> >>>>>>> As of right now, while trying to address some concerns, some >>>>>>> edits have been made in gdocs by Noble that we'll sync back into >>>>>>> confluence; so that's the only divergence. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Nov 23, 2021 at 9:09 PM Ishan Chattopadhyaya < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> > -1, I would like to see a proposal on list where it will >>>>>>> be permanently available >>>>>>> >>>>>>>> > in the archives instead of being directed to a Google doc >>>>>>> which can be edited at any point in time. >>>>>>> >>>>>>>> > This has been an incredibly difficult proposal to keep >>>>>>> track of. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> This entire SIP process has been incredibly difficult to >>>>>>> coordinate. The idea of doing this on a mail thread was a terrible one. >>>>>>> There's no archive available publicly for this list that doesn't break >>>>>>> the >>>>>>> threading functionality (I guess someone among us is using some funky >>>>>>> mail >>>>>>> client that's messing with Pony Mail). JIRA for discussion would've been >>>>>>> much better. I suppose JIRA used to have threaded discussions once upon >>>>>>> a >>>>>>> time, that would've been very handy here. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> On Tue, Nov 23, 2021 at 9:07 PM Ishan Chattopadhyaya < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>> Okay, sure, I'll rescind my assertion about having reached >>>>>>> a lazy consensus. I'll work through all the recent comments and >>>>>>> feedback. >>>>>>> >>>>>>>>> I think at the last moment, there was an edit by Noble to >>>>>>> the Google Doc that hasn't made it into the SIP document, and I'll >>>>>>> review >>>>>>> that change, the recent comments and get back here tomorrow. Other than >>>>>>> that last moment change, the SIP document is up to date. >>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>> On Tue, Nov 23, 2021 at 8:24 PM Gus Heck < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> Also, I don't know that it's defined anywhere, but I feel >>>>>>> that declaring lazy consensus on a SIP in less than a week is rushing >>>>>>> things. Some folks won't have the flexibility to address things daily >>>>>>> and >>>>>>> I've been uncomfortably pulled away from paid work trying to keep up >>>>>>> with >>>>>>> this as is. Lazy consensus should include at least one weekend (IMHO) >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> On Tue, Nov 23, 2021 at 9:49 AM Gus Heck < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>>>>> >>>>>>> >>>>>>>>>>> IMHO two things must happen before lazy consensus wait >>>>>>> period can begin: >>>>>>> >>>>>>>>>>> 1) The wiki must be updated to reflect the results of >>>>>>> discussion in the google doc. >>>>>>> >>>>>>>>>>> 2) The fact that the wiki has been updated, and that >>>>>>> you think the discussion is at an end must be posted here, preferably >>>>>>> with >>>>>>> a summary on the list. >>>>>>> >>>>>>>>>>> >>>>>>> >>>>>>>>>>> If it didn't happen on the list it didn't happen. >>>>>>> >>>>>>>>>>> >>>>>>> >>>>>>>>>>> Also the google doc has not been notifying me of >>>>>>> changes/responses. So a note here when you've responded and some time to >>>>>>> respond before the above steps are taken would be appreciated. >>>>>>> >>>>>>>>>>> >>>>>>> >>>>>>>>>>> -Gus >>>>>>> >>>>>>>>>>> >>>>>>> >>>>>>>>>>> On Tue, Nov 23, 2021 at 9:40 AM Mike Drob < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> -1, I would like to see a proposal on list where it >>>>>>> will be permanently available in the archives instead of being directed >>>>>>> to >>>>>>> a Google doc which can be edited at any point in time. >>>>>>> >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> This has been an incredibly difficult proposal to keep >>>>>>> track of. >>>>>>> >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> Mike >>>>>>> >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> On Tue, Nov 23, 2021 at 6:09 AM Ishan Chattopadhyaya < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>> This proposal has passed with lazy consensus. We can >>>>>>> proceed to the implementation phase. >>>>>>> >>>>>>>>>>>>> Thanks to everyone for feedback, esp. Gus for the >>>>>>> patience. >>>>>>> >>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>> On Sun, Nov 21, 2021 at 2:24 PM Gus Heck < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> Thanks for adding that :) and Thanks to Nobel for >>>>>>> translating my ravings :). Sorry about the acronym. Amusingly I had to >>>>>>> look >>>>>>> up gish gallop to understand what you meant :) I can assure you that I >>>>>>> will >>>>>>> never gish gallop you or anyone on the list, and I find such techniques >>>>>>> repugnant. If I'm not making sense, I'm likely covering ground too fast, >>>>>>> skipping things that are in my head but not yours, expressing too many >>>>>>> tangents (when I start nesting parentheses this is (usually) a sign I'm >>>>>>> wandering too much), or I am simply mistaken about something. Always >>>>>>> feel >>>>>>> free to ask me to explain! >>>>>>> >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> -Gus >>>>>>> >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> On Sun, Nov 21, 2021 at 12:17 AM Ishan Chattopadhyaya >>>>>>> <[email protected]> wrote: >>>>>>> >>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>> Added both your suggestions to Google Docs for >>>>>>> inline commenting and the SIP document at confluence. Thanks for the >>>>>>> feedback. >>>>>>> >>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>> On Sun, Nov 21, 2021 at 10:28 AM Ishan >>>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>>> >>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>> I'll add the lifecycle details to the document as >>>>>>> well. >>>>>>> >>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>> On Sun, 21 Nov, 2021, 9:08 am Ishan Chattopadhyaya, >>>>>>> <[email protected]> wrote: >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> Hi Gus, >>>>>>> >>>>>>>>>>>>>>>>> Thanks for bringing it up again. Initially, I >>>>>>> wasn't clear what you meant and mistook them for gish gallop. >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> These were the things I am/was confused about >>>>>>> (Noble explained it to me what you mean): >>>>>>> >>>>>>>>>>>>>>>>> - adherence to DRY via zk >>>>>>> >>>>>>>>>>>>>>>>> - runtime config info should come from zk >>>>>>> >>>>>>>>>>>>>>>>> - add dynamic features >>>>>>> >>>>>>>>>>>>>>>>> - same interaction patterns for consulting config >>>>>>> values. >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> Now, after Noble's help, I think I understand your >>>>>>> motivations. >>>>>>> >>>>>>>>>>>>>>>>> - DRY = Don't Repeat Yourself ;-) >>>>>>> >>>>>>>>>>>>>>>>> - You're suggesting that we have a standard way to >>>>>>> have role specific configurations (in ZK), so that a new role added >>>>>>> later >>>>>>> can access the configurations in a standard way >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> This is a very good suggestion, and makes complete >>>>>>> sense now. This was definitely a missing piece! >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> Here's a proposal to address that (adding it to >>>>>>> Google Docs: >>>>>>> https://docs.google.com/document/d/1hijvM1WX9u2TOUdLEkFYVCofLeJlv2MRZqe-ncobJVw/edit >>>>>>> ): >>>>>>> >>>>>>>>>>>>>>>>> - Per Role ZNode >>>>>>> >>>>>>>>>>>>>>>>> - Every role's ZNode to hold configuration data >>>>>>> (this is the key addition that you're asking for, IIUC) >>>>>>> >>>>>>>>>>>>>>>>> - Children of role ZNodes to be list of ephemeral >>>>>>> solr nodes >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> We can iterate over this on the Google Docs >>>>>>> (internal representation ZK section) with inline commenting if we need >>>>>>> to. >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> Please let us know how that sounds. >>>>>>> >>>>>>>>>>>>>>>>> Regards, >>>>>>> >>>>>>>>>>>>>>>>> Ishan >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>> On Sun, Nov 21, 2021 at 4:49 AM Gus Heck < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> It is difficult to track everything in this >>>>>>> thread for sure. Specifically, I don't feel my email of Wed, Nov 17, >>>>>>> 7:43 >>>>>>> PM (EST) has been addressed or responded to except for one difference of >>>>>>> opinion with Jan on whether or not we should prohibit the notion of any >>>>>>> role ever changing at runtime. >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> Note that I made a specific proposal for an >>>>>>> addition to the sip, regarding start up lifecycle and adherence to DRY >>>>>>> via >>>>>>> zk, in the (probably not clearly expressed) hope that we can move to a >>>>>>> general overall application principle that at runtime config info should >>>>>>> come from zk. If we follow such a principle, we will always be in a >>>>>>> position to add dynamic features without significant rework, and all >>>>>>> code >>>>>>> could hopefully reuse the same interaction patterns for consulting >>>>>>> config >>>>>>> values. >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> Neither item is addressed in the SIP currently. >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> Also I had put that out there as a single item so >>>>>>> it could be focused on (simplifying the discussion I hope), and >>>>>>> depending >>>>>>> on how discussion proceeds, I may have other follow on items. >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> -Gus >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> On Sat, Nov 20, 2021 at 8:45 AM Ishan >>>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>> On Fri, 19 Nov, 2021, 7:03 pm Ilan Ginzburg, < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> Is the request here for everybody to express >>>>>>> again the concerns >>>>>>> >>>>>>>>>>>>>>>>>>>> already expressed in this email thread and not >>>>>>> addressed? >>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>> I think the expectation now (after we've >>>>>>> expressed our intention to reach a lazy consensus) is that if someone >>>>>>> wants >>>>>>> to block this SIP from adoption, then to put forward the objections. >>>>>>> Sort >>>>>>> of like a veto. >>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> I suggest instead the authors review the >>>>>>> thread, match expressed >>>>>>> >>>>>>>>>>>>>>>>>>>> concerns with how the concern was addressed (or >>>>>>> not addressed) and >>>>>>> >>>>>>>>>>>>>>>>>>>> provide an exhaustive list. >>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>> I've summarized most of the discussion in the >>>>>>> SIP document. If you feel I could do a better job with it, please help >>>>>>> me >>>>>>> with areas of improvement. >>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> This proposal in its current form (data and >>>>>>> overseer roles) doesn't >>>>>>> >>>>>>>>>>>>>>>>>>>> offer much that can't be reasonably achieved by >>>>>>> other means. I'd find >>>>>>> >>>>>>>>>>>>>>>>>>>> much more value in making sure what is done now >>>>>>> is a solid foundation >>>>>>> >>>>>>>>>>>>>>>>>>>> for the future. >>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>> Fair enough, I understand your perspective. >>>>>>> Thanks for your feedback. >>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> Ilan >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> On Thu, Nov 18, 2021 at 11:24 PM Noble Paul < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>>>>>>> > After so many back and forth mails, I just >>>>>>> can't say who has an outstanding concern and if they are already >>>>>>> addressed >>>>>>> or not. I think the Google doc would help us get clarity on that. Please >>>>>>> take a moment to give your inputs >>>>>>> >>>>>>>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>>>>>>> > On Fri, Nov 19, 2021, 9:18 AM Ishan >>>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>>> >>>>>>>>>>>>>>>>>>>> >> Apologies, the vote hasn't passed formally >>>>>>> and I was under some confusion on the process. >>>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>>> >>>>>>>>>>>>>>>>>>>> >> I'd like to proceed with a lazy consensus >>>>>>> and proceed to the implementation phase now. >>>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>>> >>>>>>>>>>>>>>>>>>>> >> However, I would appreciate it if someone >>>>>>> wants to bring out any outstanding concerns about the SIP document. >>>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>>> >>>>>>>>>>>>>>>>>>>> >> To facilitate in-line comments, here's a >>>>>>> temporary Google Docs version of this document. >>>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>>> https://docs.google.com/document/d/1hijvM1WX9u2TOUdLEkFYVCofLeJlv2MRZqe-ncobJVw/edit?usp=sharing >>>>>>> >>>>>>>>>>>>>>>>>>>> >> (I shall copy changes back to confluence >>>>>>> eventually) >>>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>>> >>>>>>>>>>>>>>>>>>>> >> Thanks and apologies again regarding the >>>>>>> confusion with the voting, >>>>>>> >>>>>>>>>>>>>>>>>>>> >> Regards, >>>>>>> >>>>>>>>>>>>>>>>>>>> >> Ishan >>>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>>> >>>>>>>>>>>>>>>>>>>> >> On Thu, Nov 18, 2021 at 9:50 PM Ishan >>>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>>> >>>>>>>>>>>>>>>>>>>> >>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>> The SIP passed the voting phase. Thanks for >>>>>>> all for the feedback and insights. >>>>>>> >>>>>>>>>>>>>>>>>>>> >>> Looking forward to your collaboration and >>>>>>> reviews as we implement this. >>>>>>> >>>>>>>>>>>>>>>>>>>> >>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>> On Thu, Nov 18, 2021 at 9:42 PM Ishan >>>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>> > It's fine if we don't provide any >>>>>>> ability for runtime modification of roles at this time but I'm leery of >>>>>>> precluding it in the future. >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>> In future, the necessity for such a >>>>>>> facility can dictate our course of action. We cannot lay down rules >>>>>>> cast in >>>>>>> stone for functionality that we can't foresee yet. >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>> On Thu, Nov 18, 2021 at 9:40 PM Ishan >>>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>> Thanks Jan, I added both those points to >>>>>>> the SIP document in the Notes section. >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>> On Thu, Nov 18, 2021 at 7:18 PM Jan >>>>>>> Høydahl <[email protected]> wrote: >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> 18. nov. 2021 kl. 01:43 skrev Gus Heck < >>>>>>> [email protected]>: >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> 2) Roles will not be checked by loading >>>>>>> config from disk or caching disk config in memory. (zk ONLY source of >>>>>>> truth) >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> It sounds a bit backward for a local >>>>>>> node to first parse solr.node.roles, determine its local set of roles, >>>>>>> then >>>>>>> publish them to Zookeeper, and then read back its own roles from ZK. >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> Code that only needs to determine "Do I >>>>>>> have the XXX role?" or find out "What roles do I have" should be able to >>>>>>> fetch the (static) roles from some roles utility class without >>>>>>> consulting >>>>>>> ZK. >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> Code that needs to check what nodes have >>>>>>> a certain role (such as placement) would obviously need ot consult ZK. >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> Perhaps the SIP should also state some >>>>>>> Non-goals or assertions such as >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> * Roles are static and immutable (also >>>>>>> in zk) for the entire life cycle of a node >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> I also think we should state that the >>>>>>> bar for adding new roles should be high so it is not abused as any other >>>>>>> tag or label for any tiny feature. It should be reserved for >>>>>>> functionality >>>>>>> that may benefit from a dedicated set of nodes. That may be clear >>>>>>> already, >>>>>>> but you never know... >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> Jan >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> >>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>> [email protected] >>>>>>> >>>>>>>>>>>>>>>>>>>> For additional commands, e-mail: >>>>>>> [email protected] >>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>> >>>>>>>>>>>>>>>>>> http://www.needhamsoftware.com (work) >>>>>>> >>>>>>>>>>>>>>>>>> http://www.the111shift.com (play) >>>>>>> >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> -- >>>>>>> >>>>>>>>>>>>>> http://www.needhamsoftware.com (work) >>>>>>> >>>>>>>>>>>>>> http://www.the111shift.com (play) >>>>>>> >>>>>>>>>>> >>>>>>> >>>>>>>>>>> >>>>>>> >>>>>>>>>>> >>>>>>> >>>>>>>>>>> -- >>>>>>> >>>>>>>>>>> http://www.needhamsoftware.com (work) >>>>>>> >>>>>>>>>>> http://www.the111shift.com (play) >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> -- >>>>>>> >>>>>>>>>> http://www.needhamsoftware.com (work) >>>>>>> >>>>>>>>>> http://www.the111shift.com (play) >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> -- >>>>>>> >>>> http://www.needhamsoftware.com (work) >>>>>>> >>>> http://www.the111shift.com (play) >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > -- >>>>>>> > http://www.needhamsoftware.com (work) >>>>>>> > http://www.the111shift.com (play) >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>> For additional commands, e-mail: [email protected] >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> http://www.needhamsoftware.com (work) >>>>>> http://www.the111shift.com (play) >>>>>> >>>>> >>>> >>>> -- >>>> http://www.needhamsoftware.com (work) >>>> http://www.the111shift.com (play) >>>> >>>
