Thanks for trying to summarize and drive the work Ishan. I'd like to add
Scope of SIP Ishan: Role API and config Jan: Role API, config, and impact of one real role e.g. the "data" role, to examplify and justify the role infrastructure According to SIP process the next step is not implementation, but rather to iterate the SIP text to something you believe would pass a vote. It's hard to stitch together all these email and mini summaries into a meaningful whole. Jan > 15. nov. 2021 kl. 05:28 skrev Ishan Chattopadhyaya > <ichattopadhy...@gmail.com>: > > Thanks to everyone for the feedback. > > Here's an attempt to summarize broad topics discussed. > > No negative roles > Everyone agree > > Roles on/off by default? > Jason+(Ilan,Houston?): All roles to be on by default > Gus,Ishan,Noble: Only those roles to be on by default that are needed for > backcompat > > Which branch to target? > Jan,Ishan,Noble: New feature to be added to 9x branch > > Need for roles? > Tim: new concept of nodes unnecessary since everything that's proposed can be > achieved using changes to new autoscaling framework and replica placement > plugins. > Ishan,Noble: A first class concept of roles is important so that this > functionality is expected to work, irrespective of whatever custom placement > plugins users deploy (since placement plugins don't support chaining). > > Roles for collections? > Ilan: Role aware collections > Ishan: This can be implemented separately later using node roles and > placement plugins. > > Configuration > Sysprops vs solr.xml+sysprops vs envvars: > Shawn: Solr.xml and/or envvars > Houston,Ilan: Sysprops and/or envvars > Ishan,Noble: Sysprops > Jan: SIP-11 > > Outstanding issues > Shawn: Color of the bikeshed ;-) > > Please let me know if I missed something here. If there are no further strong > objections, we can proceed to the implementation phase. There's already a > draft/WIP PR in the works: https://github.com/apache/solr/pull/403 > <https://github.com/apache/solr/pull/403> > > Thanks, > Ishan > > On Fri, Nov 12, 2021 at 11:38 PM Gus Heck <gus.h...@gmail.com > <mailto:gus.h...@gmail.com>> wrote: > Yeah we should only be looking for and only be reporting (if we choose to > report to the user) a specific set of env variables. Anything else should be > ignored.Should be an enum or constants somewhere listing what solr cares > about, and we should ignore or be blind to anything else. > > Perhaps we'd like to have a ConfigParams (or whatever) enum that has methods > returning the env, sysprop, bin/solr arg, configFile and zkLocation that can > be used to provide each possible configuration option (for things that are > single value or short list, obviously an entire schema probably would not be > setable by sysprop :) )? > > The return type of those methods could be Optional<>() since we neither have > all of those for everything any time soon, and not all of them will make > sense in all cases. > > zkLocation is a bit tricky and nebulous since it's probably a zk path and a > JSON path or Xpath combined and relative to the chroot which itself is a > potential config param, some stuff to think through there. > > On Thu, Nov 11, 2021 at 3:49 PM Ilan Ginzburg <ilans...@gmail.com > <mailto:ilans...@gmail.com>> wrote: > Houston made a very valid comment back then on the placement plugin support > of environment variables (dropped as a consequence). > > https://issues.apache.org/jira/browse/SOLR-15019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286680#comment-17286680 > > <https://issues.apache.org/jira/browse/SOLR-15019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286680#comment-17286680> > It could be possible to unintentionally leak node data that should be kept > secret if Solr is allowed to freely access (random?) environment variables as > part of configuration. > > Something to keep in mind. > > Ilan > > > On Thu 11 Nov 2021 at 20:12, Eric Pugh <ep...@opensourceconnections.com > <mailto:ep...@opensourceconnections.com>> wrote: > Agreed! > > I’ve noticed that in the Play Framework, you can configure everything via a > property based configuration file, however it makes it easy to override the > property file via another one, or via an ENV variables: > > db.default.username="smui" > db.default.username=${?SMUI_DB_USER} > > Which turns out to be very liberating! > > >> On Nov 11, 2021, at 2:09 PM, Jan Høydahl <jan....@cominvent.com >> <mailto:jan....@cominvent.com>> wrote: >> >> +1 to a roundup of env and props across the board. I think SIP 11 is on the >> track of something. But can be done independent of this. >> >> Jan Høydahl >> >>> 11. nov. 2021 kl. 17:44 skrev Gus Heck <gus.h...@gmail.com >>> <mailto:gus.h...@gmail.com>>: >>> >>> >>> I guess all I mean is that it shouldn't be only sysprops. Enabling >>> sysprops, Env vars etc seems fine but we need to clearly document >>> precedence among any/all options. What is convenient varies from case to >>> case and in a perfect world what I'd like to see is full support across >>> each style (files, zk, props, env vars) with consistent and obvious naming >>> and well documented resolution order. >>> >>> What I don't like is a little bit of env vars for some stuff, props for >>> others, files for yet more stuff and some unclear aggregation of that >>> showing up in zk... (or maybe some of it not showing up anywhere code could >>> check it...) >>> >>> On Thu, Nov 11, 2021 at 11:07 AM Houston Putman <houstonput...@gmail.com >>> <mailto:houstonput...@gmail.com>> wrote: >>> I agree with Jan, when thinking about making Solr as cloud friendly as >>> possible EnvVars and (to a lesser extent) sysProps are much preferable than >>> having a setting in the solr.xml. >>> This is because it's easier to customize EnvVars per-node, while >>> customizing a config file is much harder, as those tend to be static and >>> shared across a whole environment. >>> >>> Also thanks for linking that SIP Jan, very applicable. >>> >>> - Houston >>> >>> On Fri, Nov 5, 2021 at 5:19 PM Jan Høydahl <jan....@cominvent.com >>> <mailto:jan....@cominvent.com>> wrote: >>> Thinking of these roles as labels, I think sysProps and envVars are the two >>> universal methods, and nothing wrong with that. >>> I keep trying to think cloud native and container, so having excellent 1st >>> class support for env.vars for such configs is a priority to me. >>> Most tools, CI-environments etc have built-in support for env.vars, and so >>> it makes sense to me. >>> >>> See >>> https://cwiki.apache.org/confluence/display/SOLR/SIP-11+Uniform+cluster-level+configuration+API >>> >>> <https://cwiki.apache.org/confluence/display/SOLR/SIP-11+Uniform+cluster-level+configuration+API> >>> for some interesting ideas around cluster/node level config. >>> >>> See >>> >>>> 5. nov. 2021 kl. 15:04 skrev Gus Heck <gus.h...@gmail.com >>>> <mailto:gus.h...@gmail.com>>: >>>> >>>> Agree better to something other than sysprops. an arg in the start script >>>> would be friendlier than -D props which generally are irritatingly verbose >>>> and expose too much implementation. >>>> >>>> We lack a config file per level. solr.xml does double duty as global and >>>> per-node depending on how it's used (zk or filesystem). >>>> >>>> Config file names are confusing too. Our file names are legacy of >>>> non-cloud mode I think, and we really should at some point (10.x?) rework >>>> configs to be cluster.xml, node.xml, collection.xml (formerly >>>> solrconfig.xml) and schema.xml (and maybe support something other than >>>> xml, but that's not nearly as important as clarity in naming, and having >>>> features) >>>> >>>> But this is all straying way off topic and should have its own SIP if >>>> someone seems to have time for it :) >>>> >>>> On Thu, Nov 4, 2021 at 6:07 PM Shawn Heisey <elyog...@elyograg.org >>>> <mailto:elyog...@elyograg.org>> wrote: >>>> On 11/4/21 2:51 PM, Noble Paul wrote: >>>> > The SIP can be boiled down to the following >>>> > >>>> > * *Tag a node with a label (role) using a system property* >>>> > ** Use the placement plugin to whitelist/block list certain nodes* >>>> > ** Publish the roles through an API* >>>> >>>> >>>> In general, for Solr, do we like the idea of having things controlled by >>>> system properties? >>>> >>>> I would think solr.xml would be the right place to configure this, >>>> except that people can and probably do put solr.xml in zookeeper, which >>>> would mean every system would have the SAME solr.xml, and we're back to >>>> system properties as a way to customize solr.xml on each system. >>>> >>>> I have never used system properties to configure Solr. When I customize >>>> the config, I will often remove property substitutions from it and go >>>> with explicit settings. My general opinion about system properties is >>>> that if they're going to be used, they should DIRECTLY configure the >>>> application, not be sent in via property substitution in a config file. >>>> I've never liked the way our default configs use that paradigm. It >>>> means you cannot look at the config and know exactly how things are >>>> configured, without finding out whether system properties have been set. >>>> >>>> What color do others think that bikeshed should be painted? >>>> >>>> Thanks, >>>> Shawn >>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >>>> <mailto:dev-unsubscr...@solr.apache.org> >>>> For additional commands, e-mail: dev-h...@solr.apache.org >>>> <mailto:dev-h...@solr.apache.org> >>>> >>>> >>>> >>>> -- >>>> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work) >>>> http://www.the111shift.com <http://www.the111shift.com/> (play) >>> >>> >>> >>> -- >>> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work) >>> http://www.the111shift.com <http://www.the111shift.com/> (play) > > _______________________ > Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | > http://www.opensourceconnections.com <http://www.opensourceconnections.com/> > | My Free/Busy <http://tinyurl.com/eric-cal> > Co-Author: Apache Solr Enterprise Search Server, 3rd Ed > <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw> > > This e-mail and all contents, including attachments, is considered to be > Company Confidential unless explicitly stated otherwise, regardless of > whether attachments are marked as such. > > > > -- > http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work) > http://www.the111shift.com <http://www.the111shift.com/> (play)
