Thanks to everyone for the feedback. Here's an attempt to summarize broad topics discussed.
*No negative roles* Everyone agree *Roles on/off by default?* Jason+(Ilan,Houston?): All roles to be on by default Gus,Ishan,Noble: Only those roles to be on by default that are needed for backcompat *Which branch to target?* Jan,Ishan,Noble: New feature to be added to 9x branch *Need for roles?* Tim: new concept of nodes unnecessary since everything that's proposed can be achieved using changes to new autoscaling framework and replica placement plugins. Ishan,Noble: A first class concept of roles is important so that this functionality is expected to work, irrespective of whatever custom placement plugins users deploy (since placement plugins don't support chaining). *Roles for collections?* Ilan: Role aware collections Ishan: This can be implemented separately later using node roles and placement plugins. *Configuration* Sysprops vs solr.xml+sysprops vs envvars: Shawn: Solr.xml and/or envvars Houston,Ilan: Sysprops and/or envvars Ishan,Noble: Sysprops Jan: SIP-11 *Outstanding issues* Shawn: Color of the bikeshed ;-) Please let me know if I missed something here. If there are no further strong objections, we can proceed to the implementation phase. There's already a draft/WIP PR in the works: https://github.com/apache/solr/pull/403 Thanks, Ishan On Fri, Nov 12, 2021 at 11:38 PM Gus Heck <gus.h...@gmail.com> wrote: > Yeah we should only be looking for and only be reporting (if we choose to > report to the user) a specific set of env variables. Anything else should > be ignored.Should be an enum or constants somewhere listing what solr cares > about, and we should ignore or be blind to anything else. > > Perhaps we'd like to have a ConfigParams (or whatever) enum that has > methods returning the env, sysprop, bin/solr arg, configFile and zkLocation > that can be used to provide each possible configuration option (for things > that are single value or short list, obviously an entire schema probably > would not be setable by sysprop :) )? > > The return type of those methods could be Optional<>() since we neither > have all of those for everything any time soon, and not all of them will > make sense in all cases. > > zkLocation is a bit tricky and nebulous since it's probably a zk path and > a JSON path or Xpath combined and relative to the chroot which itself is a > potential config param, some stuff to think through there. > > On Thu, Nov 11, 2021 at 3:49 PM Ilan Ginzburg <ilans...@gmail.com> wrote: > >> Houston made a very valid comment back then on the placement plugin >> support of environment variables (dropped as a consequence). >> >> >> https://issues.apache.org/jira/browse/SOLR-15019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286680#comment-17286680 >> >> It could be possible to unintentionally leak node data that should be >> kept secret if Solr is allowed to freely access (random?) environment >> variables as part of configuration. >> >> Something to keep in mind. >> >> Ilan >> >> >> On Thu 11 Nov 2021 at 20:12, Eric Pugh <ep...@opensourceconnections.com> >> wrote: >> >>> Agreed! >>> >>> I’ve noticed that in the Play Framework, you can configure everything >>> via a property based configuration file, however it makes it easy to >>> override the property file via another one, or via an ENV variables: >>> >>> db.default.username="smui" >>> db.default.username=${?SMUI_DB_USER} >>> >>> Which turns out to be very liberating! >>> >>> >>> On Nov 11, 2021, at 2:09 PM, Jan Høydahl <jan....@cominvent.com> wrote: >>> >>> +1 to a roundup of env and props across the board. I think SIP 11 is on >>> the track of something. But can be done independent of this. >>> >>> Jan Høydahl >>> >>> 11. nov. 2021 kl. 17:44 skrev Gus Heck <gus.h...@gmail.com>: >>> >>> >>> I guess all I mean is that it shouldn't be only sysprops. Enabling >>> sysprops, Env vars etc seems fine but we need to clearly document >>> precedence among any/all options. What is convenient varies from case to >>> case and in a perfect world what I'd like to see is full support across >>> each style (files, zk, props, env vars) with consistent and obvious naming >>> and well documented resolution order. >>> >>> What I don't like is a little bit of env vars for some stuff, props for >>> others, files for yet more stuff and some unclear aggregation of that >>> showing up in zk... (or maybe some of it not showing up anywhere code could >>> check it...) >>> >>> On Thu, Nov 11, 2021 at 11:07 AM Houston Putman <houstonput...@gmail.com> >>> wrote: >>> >>>> I agree with Jan, when thinking about making Solr as cloud friendly as >>>> possible EnvVars and (to a lesser extent) sysProps are much preferable than >>>> having a setting in the solr.xml. >>>> This is because it's easier to customize EnvVars per-node, while >>>> customizing a config file is much harder, as those tend to be static and >>>> shared across a whole environment. >>>> >>>> Also thanks for linking that SIP Jan, very applicable. >>>> >>>> - Houston >>>> >>>> On Fri, Nov 5, 2021 at 5:19 PM Jan Høydahl <jan....@cominvent.com> >>>> wrote: >>>> >>>>> Thinking of these roles as labels, I think sysProps and envVars are >>>>> the two universal methods, and nothing wrong with that. >>>>> I keep trying to think cloud native and container, so having excellent >>>>> 1st class support for env.vars for such configs is a priority to me. >>>>> Most tools, CI-environments etc have built-in support for env.vars, >>>>> and so it makes sense to me. >>>>> >>>>> See >>>>> https://cwiki.apache.org/confluence/display/SOLR/SIP-11+Uniform+cluster-level+configuration+API >>>>> for some interesting ideas around cluster/node level config. >>>>> >>>>> See >>>>> >>>>> 5. nov. 2021 kl. 15:04 skrev Gus Heck <gus.h...@gmail.com>: >>>>> >>>>> Agree better to something other than sysprops. an arg in the start >>>>> script would be friendlier than -D props which generally are irritatingly >>>>> verbose and expose too much implementation. >>>>> >>>>> We lack a config file per level. solr.xml does double duty as global >>>>> and per-node depending on how it's used (zk or filesystem). >>>>> >>>>> Config file names are confusing too. Our file names are legacy of >>>>> non-cloud mode I think, and we really should at some point (10.x?) rework >>>>> configs to be cluster.xml, node.xml, collection.xml (formerly >>>>> solrconfig.xml) and schema.xml (and maybe support something other than >>>>> xml, >>>>> but that's not nearly as important as clarity in naming, and having >>>>> features) >>>>> >>>>> But this is all straying way off topic and should have its own SIP if >>>>> someone seems to have time for it :) >>>>> >>>>> On Thu, Nov 4, 2021 at 6:07 PM Shawn Heisey <elyog...@elyograg.org> >>>>> wrote: >>>>> >>>>>> On 11/4/21 2:51 PM, Noble Paul wrote: >>>>>> > The SIP can be boiled down to the following >>>>>> > >>>>>> > * *Tag a node with a label (role) using a system property* >>>>>> > ** Use the placement plugin to whitelist/block list certain nodes* >>>>>> > ** Publish the roles through an API* >>>>>> >>>>>> >>>>>> In general, for Solr, do we like the idea of having things controlled >>>>>> by >>>>>> system properties? >>>>>> >>>>>> I would think solr.xml would be the right place to configure this, >>>>>> except that people can and probably do put solr.xml in zookeeper, >>>>>> which >>>>>> would mean every system would have the SAME solr.xml, and we're back >>>>>> to >>>>>> system properties as a way to customize solr.xml on each system. >>>>>> >>>>>> I have never used system properties to configure Solr. When I >>>>>> customize >>>>>> the config, I will often remove property substitutions from it and go >>>>>> with explicit settings. My general opinion about system properties >>>>>> is >>>>>> that if they're going to be used, they should DIRECTLY configure the >>>>>> application, not be sent in via property substitution in a config >>>>>> file. >>>>>> I've never liked the way our default configs use that paradigm. It >>>>>> means you cannot look at the config and know exactly how things are >>>>>> configured, without finding out whether system properties have been >>>>>> set. >>>>>> >>>>>> What color do others think that bikeshed should be painted? >>>>>> >>>>>> Thanks, >>>>>> Shawn >>>>>> >>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >>>>>> For additional commands, e-mail: dev-h...@solr.apache.org >>>>>> >>>>>> >>>>> >>>>> -- >>>>> http://www.needhamsoftware.com (work) >>>>> http://www.the111shift.com (play) >>>>> >>>>> >>>>> >>> >>> -- >>> http://www.needhamsoftware.com (work) >>> http://www.the111shift.com (play) >>> >>> >>> _______________________ >>> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467 >>> | http://www.opensourceconnections.com | My Free/Busy >>> <http://tinyurl.com/eric-cal> >>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed >>> <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw> >>> This e-mail and all contents, including attachments, is considered to be >>> Company Confidential unless explicitly stated otherwise, regardless >>> of whether attachments are marked as such. >>> >>> > > -- > http://www.needhamsoftware.com (work) > http://www.the111shift.com (play) >
