+1
On Thu, Nov 25, 2021, 6:02 PM Ishan Chattopadhyaya < [email protected]> wrote: > I've started a formal vote thread for the proposal here. Thanks all. > > On Thu, Nov 25, 2021 at 12:10 PM Ishan Chattopadhyaya < > [email protected]> wrote: > >> We have closed the Google Doc for further comments, and all changes have >> been incorporated into the confluence document: >> https://cwiki.apache.org/confluence/display/SOLR/SIP-15+Node+roles >> Thanks for all the comments and feedback. >> >> On Thu, Nov 25, 2021 at 12:09 PM Ishan Chattopadhyaya < >> [email protected]> wrote: >> >>> > Maybe one way to provide for the future is to place some requirements >>> on roles, and designate that role-specific coordination information >>> > be placed within the subtree for that role under >>> /node-roles/<rolename>/ To facilitate this we could >>> > 1. move the currently designed set of ephemeral nodes down one level >>> into am "/available" node that reflects which nodes are currently up with >>> the role (exactly what they do in the present design) >>> >>> We've introduced nesting in the ZK structure now. Ephemeral nodes for >>> Solr node names will now be placed at >>> /node-roles/<rolename>/nodes/<ephemeralnode>. >>> >>> >>> >>> >>> >>> >>> On Thu, Nov 25, 2021 at 12:04 PM Ishan Chattopadhyaya < >>> [email protected]> wrote: >>> >>>> >>>> >>>> On Thu, Nov 25, 2021 at 4:35 AM Gus Heck <[email protected]> wrote: >>>> >>>>> Things I think still need to be clarified to define "Role" explicitly. >>>>> Primarily to guide future implementations. Things I can think of include: >>>>> >>>>> - What the bar for something to be a role vs being more >>>>> appropriate as a property/tag type concept. Essentially what is and is >>>>> not >>>>> expected (at this time) to be a role. >>>>> >>>>> We don't wish to set a bar for anything at this point, since I can't >>>> imagine all future innovation that can happen. All we can do is present >>>> representative examples. Common sense will dictate what is useful to have >>>> as a role, and clearly we can't set a bar for that. >>>> >>>> >>>>> >>>>> - What's the process for this? Do we want a SIP per role to ensure >>>>> good discussion? Are there some simple things a role might do that are >>>>> "normal" and don't require a SIP? Or maybe no sip generally? >>>>> >>>>> The community can decide on a case by case basis using established >>>> community processes. I don't see the need to define a new process for this. >>>> >>>> >>>>> >>>>> - Where a role is defined (enum? class? which module? core? >>>>> solrj?) and what motivates that choice (specifics such as >>>>> methods/properties can be left to impl of course) >>>>> >>>>> This can be discussed during the implementation phase. This SIP is for >>>> the general concept, public interface and other non-code concerns. I don't >>>> want to start another 20 page thread here debating enum vs class and other >>>> things that might tickle someone's fancy. Lets cross the bridge once we get >>>> to it. >>>> >>>> >>>>> >>>>> - Naming conventions for roles... camel case? snake case? all >>>>> caps? what characters are allowed, Possibly discourage any ideas with >>>>> role >>>>> names that embed config options such as overseer_priority_1 and any >>>>> parsing >>>>> of role names. >>>>> >>>>> >>>> We can discuss that on a case by case basis when new roles are >>>> introduced. I've already proposed that the sysprops for roles should be in >>>> lowercase, which rules out camelCase. >>>> >>>> >>>>> >>>>> - >>>>> >>>>> If we answer those and bundle them with the 3 sections under the >>>>> example api calls into a "Roles" section or "Definition of Roles" or >>>>> something like that it seems good to me. >>>>> >>>>> Sure we could discuss any of this independently for each role in each >>>>> sip but if we put guidance here we probably can skip that discussion for N >>>>> out of M sips/tickets in the future (N being maximized if we choose well >>>>> and write with clarity). Thus we'd be saving time in the long run and >>>>> reduce the possibility that people propose something awkward and it slips >>>>> through on lazy consensus. >>>>> >>>>> And of course it's Apache software so it can all be changed later by a >>>>> vote or concensus. :) >>>>> >>>>> -Gus >>>>> >>>>> On Wed, Nov 24, 2021 at 3:21 PM Ishan Chattopadhyaya < >>>>> [email protected]> wrote: >>>>> >>>>>> > Moreover I think it's a very role specific notion, that is not a >>>>>> good >>>>>> > fit in the roles framework, so maybe should never be part of this >>>>>> SIP. >>>>>> >>>>>> Regarding "capable" vs "currently providing", I agree with Ilan here ^ >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Nov 24, 2021 at 10:38 PM Gus Heck <[email protected]> wrote: >>>>>> >>>>>>> You do make a good point that the details can be very role specific. >>>>>>> Maybe one way to provide for the future is to place some requirements on >>>>>>> roles, and designate that role-specific coordination information be >>>>>>> placed >>>>>>> within the subtree for that role under /node-roles/<rolename>/ To >>>>>>> facilitate this we could >>>>>>> >>>>>>> 1. move the currently designed set of ephemeral nodes down one >>>>>>> level into am "/available" node that reflects which nodes are >>>>>>> currently up >>>>>>> with the role (exactly what they do in the present design) >>>>>>> 2. Roles must store role related info in the data for the role >>>>>>> node, or in tree(s) that are peers to /available as they see fit. >>>>>>> 3. Roles must have documentation detailing what they store where >>>>>>> and how it's used. >>>>>>> 4. Roles (beyond those implemented in this SIP) should be >>>>>>> discussed in their own SIP (enforcing the non-trivial label concept >>>>>>> we all >>>>>>> seem to agree on) >>>>>>> >>>>>>> >>>>>> I am not in favour of complicating the currently proposed design by >>>>>> doing any of this now ^ If we need to, we can discuss this then. >>>>>> >>>>>> >>>>>>> The bit that's not clear is where that documentation would live. >>>>>>> It's not really material for the users, but more for the developers. >>>>>>> >>>>>>> On Wed, Nov 24, 2021 at 11:11 AM Ilan Ginzburg <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> I agree with Ishan that "currently providing" shouldn't be a current >>>>>>>> concern for the SIP. >>>>>>>> Moreover I think it's a very role specific notion, that is not a >>>>>>>> good >>>>>>>> fit in the roles framework, so maybe should never be part of this >>>>>>>> SIP. >>>>>>>> >>>>>>>> Even on the Overseer example. Suppose we changed the current design >>>>>>>> to >>>>>>>> have a "main" node being Overseer and a secondary to be somewhat >>>>>>>> active standby in case the main one goes down (imagine some form of >>>>>>>> replication between the two?). Both would have the "Overseer" role, >>>>>>>> but what they provide would be different. Overseer specific code >>>>>>>> should be able to know what is provided by which node, because a >>>>>>>> node >>>>>>>> trying to use the Overseer service would have to talk to a specific >>>>>>>> one of the two, not just one that currently provides. >>>>>>>> >>>>>>>> Same for data nodes. A data node can have replicas on it. But it may >>>>>>>> not. If it doesn't, shall we say it has the capability but does not >>>>>>>> provide? Of course not, nobody would suggest that. Because we know >>>>>>>> that tracking the actual data is not a role level responsibility >>>>>>>> but a >>>>>>>> data (collection/shard/replica etc) tracking responsibility, done >>>>>>>> using all the nice structures the code maintains in ZK and in caches >>>>>>>> all over the place. >>>>>>>> >>>>>>>> Ilan >>>>>>>> >>>>>>>> On Wed, Nov 24, 2021 at 5:10 PM Gus Heck <[email protected]> >>>>>>>> wrote: >>>>>>>> > >>>>>>>> > So we are potentially talking past each other. Let me re-clarify >>>>>>>> since you've interpreted "configuration" differently (I think). What >>>>>>>> I'm >>>>>>>> referring to in ZK is the "reflection of configuration" not the actual >>>>>>>> configuration. This is why I'm asking the question of how much info >>>>>>>> should >>>>>>>> be retained about a node after it goes down. Case in point zookeeper >>>>>>>> embedded: The code might take different actions if it can see that a >>>>>>>> node >>>>>>>> that had a running zookeeper did exist (and thus may return) vs a >>>>>>>> situation >>>>>>>> where there's only 2 nodes playing zookeeper roles right now and no >>>>>>>> evidence of a third ever having existed. >>>>>>>> > >>>>>>>> > Also you say you can't imagine anything beyond overseer... well I >>>>>>>> could imagine that a cluster would want to say these 10 machines (but >>>>>>>> not >>>>>>>> the other 40) are allowed to have zookeeper, and I want the cluster to >>>>>>>> to >>>>>>>> have 3 node level of redundancy in zookeeper, and a new zookeeper >>>>>>>> should be >>>>>>>> created and the data replicated to it if a zk node is lost for more >>>>>>>> than 30 >>>>>>>> minutes. >>>>>>>> > >>>>>>>> > On Wed, Nov 24, 2021 at 10:38 AM Ishan Chattopadhyaya < >>>>>>>> [email protected]> wrote: >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> On Wed, Nov 24, 2021 at 9:05 PM Ishan Chattopadhyaya < >>>>>>>> [email protected]> wrote: >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> On Wed, Nov 24, 2021 at 8:19 PM Gus Heck <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>> >>>>>>>> >>>> Left some additional comments in the google doc, extracting >>>>>>>> the key point for others here so they can know if they want to read the >>>>>>>> google doc commentary: >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> I'm addressing them here, inline. >>>>>>>> >>> >>>>>>>> >>>> >>>>>>>> >>>> There is presently disagreement with my proposal that the sip >>>>>>>> specify both how we track configuration (which I have been referring >>>>>>>> to as >>>>>>>> "capability") and runtime state ("providing"). I see the primary value >>>>>>>> of >>>>>>>> having a role concept as one of organizing the code and configuration >>>>>>>> in a >>>>>>>> way that is easy to understand, which is why I'm in favor of this SiP >>>>>>>> overall. >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> I purposely don't want to complicate this design by allowing >>>>>>>> for "capability" vs "currently providing" to be represented as first >>>>>>>> class >>>>>>>> citizens. Besides the OVERSEER role (preferred overseer), I can't >>>>>>>> think of >>>>>>>> concrete scenarios where such a concept could be applicable, and hence >>>>>>>> I >>>>>>>> don't want to get ahead of myself in trying to address how to solve >>>>>>>> that. >>>>>>>> Since the current proposal for dealing with roles is flexible and >>>>>>>> generic >>>>>>>> enough, I don't think adding such extensions would be a problem. >>>>>>>> However, I >>>>>>>> don't want to go down that rabbithole at this point in time. >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> If you don't want to tackle implementing the runtime state bit >>>>>>>> I'm ok with that as long as we have a clear plan in the SIP of how this >>>>>>>> type of information should be organized in the future. >>>>>>>> >>>> The structure proposed in the SIP for information in zk seems >>>>>>>> to map more clearly to runtime state than configuration, and seems >>>>>>>> like it >>>>>>>> would inhibit a natural representation of such. >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> The structure proposed allows for role specific configurations >>>>>>>> (if needed). It doesn't provide for node specific configuration in ZK, >>>>>>>> since all node properties should come from either solr.xml or >>>>>>>> sysprops. At >>>>>>>> present, I don't think we ever have node specific info configuration >>>>>>>> in ZK >>>>>>>> in any part of Solr, and I think we shouldn't go that route here. >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> Correction: "At present, I don't think we ever have node >>>>>>>> specific configuration in ZK in any part of Solr, and I think we >>>>>>>> shouldn't >>>>>>>> go that route here." >>>>>>>> >> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> Finally the structure in the SIP for zk information doesn't >>>>>>>> specify if some of the nodes are ephemeral, and reliable for a >>>>>>>> "current" >>>>>>>> list or if they are persistent over time. I feel like we may not yet >>>>>>>> have a >>>>>>>> plan for how much information about nodes (n general) should be >>>>>>>> retained >>>>>>>> while a node is down, which maybe needs to underlie this design. >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> All the node names under the roles znodes are ephemeral. Added >>>>>>>> this to the document (confluence and gdoc both). >>>>>>>> >>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> -Gus >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> On Tue, Nov 23, 2021 at 10:34 PM Ishan Chattopadhyaya < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>> >>>>>>>> >>>>> I've synced back all changes to confluence. Both represent >>>>>>>> the same document as of now. >>>>>>>> >>>>> Please review and suggest if there are any outstanding >>>>>>>> concerns. Thanks for all the feedback. >>>>>>>> >>>>> >>>>>>>> >>>>> I wish to proceed with the implementation based on lazy >>>>>>>> consensus again. >>>>>>>> >>>>> >>>>>>>> >>>>> On Tue, Nov 23, 2021 at 9:38 PM Mike Drob <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>> >>>>>>>> >>>>>> I should clarify, that my -1 was specifically about reaching >>>>>>>> lazy consensus and not about that proposal itself. >>>>>>>> >>>>>> >>>>>>>> >>>>>> On Tue, Nov 23, 2021 at 9:42 AM Ishan Chattopadhyaya < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> > -1, I would like to see a proposal on list where it will >>>>>>>> be permanently available >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> It will be available in the confluence document (as it >>>>>>>> currently is). The google docs is just for collecting temporary >>>>>>>> feedback, >>>>>>>> thanks to easy inline commenting capability. We promised to consolidate >>>>>>>> feedback in the gdoc and sync back to the SIP document in confluence. >>>>>>>> This >>>>>>>> is reasonable enough expectations, and shouldn't be a worry enough for >>>>>>>> a -1. >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> As of right now, while trying to address some concerns, >>>>>>>> some edits have been made in gdocs by Noble that we'll sync back into >>>>>>>> confluence; so that's the only divergence. >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> On Tue, Nov 23, 2021 at 9:09 PM Ishan Chattopadhyaya < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> > -1, I would like to see a proposal on list where it will >>>>>>>> be permanently available >>>>>>>> >>>>>>>> > in the archives instead of being directed to a Google >>>>>>>> doc which can be edited at any point in time. >>>>>>>> >>>>>>>> > This has been an incredibly difficult proposal to keep >>>>>>>> track of. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> This entire SIP process has been incredibly difficult to >>>>>>>> coordinate. The idea of doing this on a mail thread was a terrible one. >>>>>>>> There's no archive available publicly for this list that doesn't break >>>>>>>> the >>>>>>>> threading functionality (I guess someone among us is using some funky >>>>>>>> mail >>>>>>>> client that's messing with Pony Mail). JIRA for discussion would've >>>>>>>> been >>>>>>>> much better. I suppose JIRA used to have threaded discussions once >>>>>>>> upon a >>>>>>>> time, that would've been very handy here. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Nov 23, 2021 at 9:07 PM Ishan Chattopadhyaya < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> Okay, sure, I'll rescind my assertion about having >>>>>>>> reached a lazy consensus. I'll work through all the recent comments and >>>>>>>> feedback. >>>>>>>> >>>>>>>>> I think at the last moment, there was an edit by Noble to >>>>>>>> the Google Doc that hasn't made it into the SIP document, and I'll >>>>>>>> review >>>>>>>> that change, the recent comments and get back here tomorrow. Other than >>>>>>>> that last moment change, the SIP document is up to date. >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> On Tue, Nov 23, 2021 at 8:24 PM Gus Heck < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> Also, I don't know that it's defined anywhere, but I >>>>>>>> feel that declaring lazy consensus on a SIP in less than a week is >>>>>>>> rushing >>>>>>>> things. Some folks won't have the flexibility to address things daily >>>>>>>> and >>>>>>>> I've been uncomfortably pulled away from paid work trying to keep up >>>>>>>> with >>>>>>>> this as is. Lazy consensus should include at least one weekend (IMHO) >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> On Tue, Nov 23, 2021 at 9:49 AM Gus Heck < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> IMHO two things must happen before lazy consensus wait >>>>>>>> period can begin: >>>>>>>> >>>>>>>>>>> 1) The wiki must be updated to reflect the results of >>>>>>>> discussion in the google doc. >>>>>>>> >>>>>>>>>>> 2) The fact that the wiki has been updated, and that >>>>>>>> you think the discussion is at an end must be posted here, preferably >>>>>>>> with >>>>>>>> a summary on the list. >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> If it didn't happen on the list it didn't happen. >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> Also the google doc has not been notifying me of >>>>>>>> changes/responses. So a note here when you've responded and some time >>>>>>>> to >>>>>>>> respond before the above steps are taken would be appreciated. >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> -Gus >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> On Tue, Nov 23, 2021 at 9:40 AM Mike Drob < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> -1, I would like to see a proposal on list where it >>>>>>>> will be permanently available in the archives instead of being >>>>>>>> directed to >>>>>>>> a Google doc which can be edited at any point in time. >>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> This has been an incredibly difficult proposal to keep >>>>>>>> track of. >>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> Mike >>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> On Tue, Nov 23, 2021 at 6:09 AM Ishan Chattopadhyaya < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>> This proposal has passed with lazy consensus. We can >>>>>>>> proceed to the implementation phase. >>>>>>>> >>>>>>>>>>>>> Thanks to everyone for feedback, esp. Gus for the >>>>>>>> patience. >>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>> On Sun, Nov 21, 2021 at 2:24 PM Gus Heck < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> Thanks for adding that :) and Thanks to Nobel for >>>>>>>> translating my ravings :). Sorry about the acronym. Amusingly I had to >>>>>>>> look >>>>>>>> up gish gallop to understand what you meant :) I can assure you that I >>>>>>>> will >>>>>>>> never gish gallop you or anyone on the list, and I find such techniques >>>>>>>> repugnant. If I'm not making sense, I'm likely covering ground too >>>>>>>> fast, >>>>>>>> skipping things that are in my head but not yours, expressing too many >>>>>>>> tangents (when I start nesting parentheses this is (usually) a sign I'm >>>>>>>> wandering too much), or I am simply mistaken about something. Always >>>>>>>> feel >>>>>>>> free to ask me to explain! >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> -Gus >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> On Sun, Nov 21, 2021 at 12:17 AM Ishan >>>>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>>>> >>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>> Added both your suggestions to Google Docs for >>>>>>>> inline commenting and the SIP document at confluence. Thanks for the >>>>>>>> feedback. >>>>>>>> >>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>> On Sun, Nov 21, 2021 at 10:28 AM Ishan >>>>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> I'll add the lifecycle details to the document as >>>>>>>> well. >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> On Sun, 21 Nov, 2021, 9:08 am Ishan >>>>>>>> Chattopadhyaya, <[email protected]> wrote: >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> Hi Gus, >>>>>>>> >>>>>>>>>>>>>>>>> Thanks for bringing it up again. Initially, I >>>>>>>> wasn't clear what you meant and mistook them for gish gallop. >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> These were the things I am/was confused about >>>>>>>> (Noble explained it to me what you mean): >>>>>>>> >>>>>>>>>>>>>>>>> - adherence to DRY via zk >>>>>>>> >>>>>>>>>>>>>>>>> - runtime config info should come from zk >>>>>>>> >>>>>>>>>>>>>>>>> - add dynamic features >>>>>>>> >>>>>>>>>>>>>>>>> - same interaction patterns for consulting config >>>>>>>> values. >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> Now, after Noble's help, I think I understand >>>>>>>> your motivations. >>>>>>>> >>>>>>>>>>>>>>>>> - DRY = Don't Repeat Yourself ;-) >>>>>>>> >>>>>>>>>>>>>>>>> - You're suggesting that we have a standard way >>>>>>>> to have role specific configurations (in ZK), so that a new role added >>>>>>>> later can access the configurations in a standard way >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> This is a very good suggestion, and makes >>>>>>>> complete sense now. This was definitely a missing piece! >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> Here's a proposal to address that (adding it to >>>>>>>> Google Docs: >>>>>>>> https://docs.google.com/document/d/1hijvM1WX9u2TOUdLEkFYVCofLeJlv2MRZqe-ncobJVw/edit >>>>>>>> ): >>>>>>>> >>>>>>>>>>>>>>>>> - Per Role ZNode >>>>>>>> >>>>>>>>>>>>>>>>> - Every role's ZNode to hold configuration data >>>>>>>> (this is the key addition that you're asking for, IIUC) >>>>>>>> >>>>>>>>>>>>>>>>> - Children of role ZNodes to be list of >>>>>>>> ephemeral solr nodes >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> We can iterate over this on the Google Docs >>>>>>>> (internal representation ZK section) with inline commenting if we need >>>>>>>> to. >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> Please let us know how that sounds. >>>>>>>> >>>>>>>>>>>>>>>>> Regards, >>>>>>>> >>>>>>>>>>>>>>>>> Ishan >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>> On Sun, Nov 21, 2021 at 4:49 AM Gus Heck < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> It is difficult to track everything in this >>>>>>>> thread for sure. Specifically, I don't feel my email of Wed, Nov 17, >>>>>>>> 7:43 >>>>>>>> PM (EST) has been addressed or responded to except for one difference >>>>>>>> of >>>>>>>> opinion with Jan on whether or not we should prohibit the notion of any >>>>>>>> role ever changing at runtime. >>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> Note that I made a specific proposal for an >>>>>>>> addition to the sip, regarding start up lifecycle and adherence to DRY >>>>>>>> via >>>>>>>> zk, in the (probably not clearly expressed) hope that we can move to a >>>>>>>> general overall application principle that at runtime config info >>>>>>>> should >>>>>>>> come from zk. If we follow such a principle, we will always be in a >>>>>>>> position to add dynamic features without significant rework, and all >>>>>>>> code >>>>>>>> could hopefully reuse the same interaction patterns for consulting >>>>>>>> config >>>>>>>> values. >>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> Neither item is addressed in the SIP currently. >>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> Also I had put that out there as a single item >>>>>>>> so it could be focused on (simplifying the discussion I hope), and >>>>>>>> depending on how discussion proceeds, I may have other follow on items. >>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> -Gus >>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> On Sat, Nov 20, 2021 at 8:45 AM Ishan >>>>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>> On Fri, 19 Nov, 2021, 7:03 pm Ilan Ginzburg, < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> Is the request here for everybody to express >>>>>>>> again the concerns >>>>>>>> >>>>>>>>>>>>>>>>>>>> already expressed in this email thread and not >>>>>>>> addressed? >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>> I think the expectation now (after we've >>>>>>>> expressed our intention to reach a lazy consensus) is that if someone >>>>>>>> wants >>>>>>>> to block this SIP from adoption, then to put forward the objections. >>>>>>>> Sort >>>>>>>> of like a veto. >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> I suggest instead the authors review the >>>>>>>> thread, match expressed >>>>>>>> >>>>>>>>>>>>>>>>>>>> concerns with how the concern was addressed >>>>>>>> (or not addressed) and >>>>>>>> >>>>>>>>>>>>>>>>>>>> provide an exhaustive list. >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>> I've summarized most of the discussion in the >>>>>>>> SIP document. If you feel I could do a better job with it, please help >>>>>>>> me >>>>>>>> with areas of improvement. >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> This proposal in its current form (data and >>>>>>>> overseer roles) doesn't >>>>>>>> >>>>>>>>>>>>>>>>>>>> offer much that can't be reasonably achieved >>>>>>>> by other means. I'd find >>>>>>>> >>>>>>>>>>>>>>>>>>>> much more value in making sure what is done >>>>>>>> now is a solid foundation >>>>>>>> >>>>>>>>>>>>>>>>>>>> for the future. >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>> Fair enough, I understand your perspective. >>>>>>>> Thanks for your feedback. >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> Ilan >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> On Thu, Nov 18, 2021 at 11:24 PM Noble Paul < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>>>>>>> > After so many back and forth mails, I just >>>>>>>> can't say who has an outstanding concern and if they are already >>>>>>>> addressed >>>>>>>> or not. I think the Google doc would help us get clarity on that. >>>>>>>> Please >>>>>>>> take a moment to give your inputs >>>>>>>> >>>>>>>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>>>>>>> > On Fri, Nov 19, 2021, 9:18 AM Ishan >>>>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >> Apologies, the vote hasn't passed formally >>>>>>>> and I was under some confusion on the process. >>>>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >> I'd like to proceed with a lazy consensus >>>>>>>> and proceed to the implementation phase now. >>>>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >> However, I would appreciate it if someone >>>>>>>> wants to bring out any outstanding concerns about the SIP document. >>>>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >> To facilitate in-line comments, here's a >>>>>>>> temporary Google Docs version of this document. >>>>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>> https://docs.google.com/document/d/1hijvM1WX9u2TOUdLEkFYVCofLeJlv2MRZqe-ncobJVw/edit?usp=sharing >>>>>>>> >>>>>>>>>>>>>>>>>>>> >> (I shall copy changes back to confluence >>>>>>>> eventually) >>>>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >> Thanks and apologies again regarding the >>>>>>>> confusion with the voting, >>>>>>>> >>>>>>>>>>>>>>>>>>>> >> Regards, >>>>>>>> >>>>>>>>>>>>>>>>>>>> >> Ishan >>>>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >> On Thu, Nov 18, 2021 at 9:50 PM Ishan >>>>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>> The SIP passed the voting phase. Thanks >>>>>>>> for all for the feedback and insights. >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>> Looking forward to your collaboration and >>>>>>>> reviews as we implement this. >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>> On Thu, Nov 18, 2021 at 9:42 PM Ishan >>>>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>> > It's fine if we don't provide any >>>>>>>> ability for runtime modification of roles at this time but I'm leery of >>>>>>>> precluding it in the future. >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>> In future, the necessity for such a >>>>>>>> facility can dictate our course of action. We cannot lay down rules >>>>>>>> cast in >>>>>>>> stone for functionality that we can't foresee yet. >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>> On Thu, Nov 18, 2021 at 9:40 PM Ishan >>>>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>> Thanks Jan, I added both those points to >>>>>>>> the SIP document in the Notes section. >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>> On Thu, Nov 18, 2021 at 7:18 PM Jan >>>>>>>> Høydahl <[email protected]> wrote: >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> 18. nov. 2021 kl. 01:43 skrev Gus Heck < >>>>>>>> [email protected]>: >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> 2) Roles will not be checked by loading >>>>>>>> config from disk or caching disk config in memory. (zk ONLY source of >>>>>>>> truth) >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> It sounds a bit backward for a local >>>>>>>> node to first parse solr.node.roles, determine its local set of roles, >>>>>>>> then >>>>>>>> publish them to Zookeeper, and then read back its own roles from ZK. >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> Code that only needs to determine "Do I >>>>>>>> have the XXX role?" or find out "What roles do I have" should be able >>>>>>>> to >>>>>>>> fetch the (static) roles from some roles utility class without >>>>>>>> consulting >>>>>>>> ZK. >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> Code that needs to check what nodes >>>>>>>> have a certain role (such as placement) would obviously need ot >>>>>>>> consult ZK. >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> Perhaps the SIP should also state some >>>>>>>> Non-goals or assertions such as >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> * Roles are static and immutable (also >>>>>>>> in zk) for the entire life cycle of a node >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> I also think we should state that the >>>>>>>> bar for adding new roles should be high so it is not abused as any >>>>>>>> other >>>>>>>> tag or label for any tiny feature. It should be reserved for >>>>>>>> functionality >>>>>>>> that may benefit from a dedicated set of nodes. That may be clear >>>>>>>> already, >>>>>>>> but you never know... >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> Jan >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> >>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>>> [email protected] >>>>>>>> >>>>>>>>>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>> [email protected] >>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>> >>>>>>>>>>>>>>>>>> http://www.needhamsoftware.com (work) >>>>>>>> >>>>>>>>>>>>>>>>>> http://www.the111shift.com (play) >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>> >>>>>>>>>>>>>> http://www.needhamsoftware.com (work) >>>>>>>> >>>>>>>>>>>>>> http://www.the111shift.com (play) >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> -- >>>>>>>> >>>>>>>>>>> http://www.needhamsoftware.com (work) >>>>>>>> >>>>>>>>>>> http://www.the111shift.com (play) >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> -- >>>>>>>> >>>>>>>>>> http://www.needhamsoftware.com (work) >>>>>>>> >>>>>>>>>> http://www.the111shift.com (play) >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> -- >>>>>>>> >>>> http://www.needhamsoftware.com (work) >>>>>>>> >>>> http://www.the111shift.com (play) >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > -- >>>>>>>> > http://www.needhamsoftware.com (work) >>>>>>>> > http://www.the111shift.com (play) >>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> http://www.needhamsoftware.com (work) >>>>>>> http://www.the111shift.com (play) >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> http://www.needhamsoftware.com (work) >>>>> http://www.the111shift.com (play) >>>>> >>>>
