Lets see, but to some extent its obvious that we need a managing component where things come together... Romain Manni-Bucau <[email protected]> schrieb am So., 28. Dez. 2014 um 11:52:
> If we want such named configs we need a ConfigManager IMO exactly like in > JCache. > Le 28 déc. 2014 00:04, "Anatole Tresch" <[email protected]> a écrit : > > > *Hi Mark* > > > > *see inline...* > > > > *Good night* > > *Anatole* > > ** > > > > 2014-12-27 21:01 GMT+01:00 Mark Struberg <[email protected]>: > > > > > > I see a chance that we are aligning now our ideas more and more ;) > > > > > > Yes, I see a lot or progress as well (on both sides). > > > > > > > > > > Basically you have to register PropertySourceProviderSpi's > > > Do you really need to? In DeltaSpike we just use the ServiceLoader > > > mechanism (has good support in SE, EE and even OSGi). > > > > > *-> That is what I meant, you still have to add a file with the given fq > > interface name under META-INF/services and add your entries. But this > > mechanisms fits for most areas ;) * > > > > > > > > In DeltaSpike the ConfigSourceProvider is actually optional. It is > there > > > for cases where you need to register multiple ConfigSources (e.g. > because > > > you have multiple property or ini files with the same name in various > > > JARs). Usually you don't use this but instead just natively register a > > > ConfigSource itself via the ServiceLoader mechanism. In that case there > > is > > > not a single line of code at all to register it. > > > > > > We don't need CDI nor anything else beyond pure Java. > > > > > *Theoretically, if people want to deploy things into the configuration > > part, which are loaded/needed, when CDI is available, they could also use > > CDI. Via BeanManager we have access to Instace and therefore we can read > > what is there. But it is a bit weired and using pure ServiceLoader is > > definitivly a good choise ;) * > > > > > > > Even when you have CDI you usually like to initialize the configuration > > > system _before_ you even start the CDI container. Otherwise we would > not > > be > > > able to utilize the config system for @Exclude-ing beans, deactivating > > > Extensions, etc. > > > > > > Thus you _always_ need to rely on the ServiceLoader mechanism. > > > > > *Agree, I never wanted to replace it, see above ;) * > > > > > > > > > > > > So for individual property sources (the ones != default) there > > > > > > > is no issue I think, because prioritization is not an issue > > > > > > Oh I think there is. I like to e.g. provide a default configuration for > > my > > > document archive connector. For easier maintenance I like to put them > > into > > > an own documentarchive.properties file which is packaged into my > > > documentarchive-connector.jar. This ConfigSource needs to get a > priority > > > which is higher than the very default ini and property files but lower > > than > > > the environment and system properties. > > > > > *My idea here was to register my own DocumentArchive > > propertysource/config, which is reading the custom property files and add > > defaults and overridings as useful by combining with the other (existing) > > property sources, e.g.* > > > > *PropertySource ps = > > > > PropertySourceBuilder.of("DocumentArchive"). > addProviders("system/default-all").addPaths("classpath: > META-INF/cfg/documentarchive.properties").addEnvironmentProperties(). > addSystemProperties().build();* > > > > *This configuration then would registered using the > > PropertySourceProviderSpi with ServiceLoader as top level property source > > and be accessible by calling:* > > > > *Configuration docCfg = Configuration.current("DocumentArchive");* > > > > > > > > > > > > Another use case would be to register a cluster wide ConfigSource which > > > reads from one of my application tables from the database. We've seen > > this > > > quite a few times in bigger projects. > > > > *For me having such a cluster-wide config in place is something that not > > the application-programme decides. It's the architecture that defines the > > mechanism and engineers it with Ops/Devs so it is working. When we have a > > usable environment it should be possible to evaluate the correct > > applicationId (and what else is required) which then can be used to > > connect to the config DB.* > > > > > > > >Level 1: globally the hierarchy of property sources, e.g. > > > > > > > > > > > >1. Boot Config, > > > >2. System config, > > > >3. Ear Config, > > > >4. App Config > > > >5. SaaS Config, > > > > > > This is a possible hierarchy dimension. But it is just ONE out of many. > > > Actually it's mixture between 2 different dimensions > > > a.) the time dimension:boot vs system (init) vs app (runtime) > > > b.) the modularity dimension: cluster vs container vs ear vs app > > > > > *Agree. Therefore we must have a flexible design, so basically every > kind > > of levels can be implemented. If we can identify hierarchies that have > > shown to be common, we might think on providing them/or parts of them as > > extension modules. But I am not sure, if we really get there. > Definitively > > it is not a first prio stuff (adding the extensions), though we must have > > full fledged example of similar complexity to bullet-prove/show-case our > > ideas / design ;)* > > > > > > > > Whether a documentarchive.endpoint.url is needed at boot time or at > > > runtime is an implementation detail of my documentarchive-connector (to > > > stay at this example). > > > > > > > *Yep. But this max affect the location, where it must be configured. If > it > > is required during boot-time the configuration must of course also be > > available at that time ;)* > > > > Back when doing CODI (and then later again at DeltaSpike) we did discuss > > > this many times and found no good solution other than make a step back > > and > > > ignore _why_ people want the one over the other. The only thing we can > do > > > is to make sure that people (programmers, admin folks, etc) have a way > to > > > define the order in exactly the way they need. They don't even need a > way > > > to redefine the priority of the 'standard' ConfigSources (env, jndi, > > etc). > > > Instead they can just add their own ConfigSource with a higher ordinal > > and > > > be done. > > > > *Agree. Would state exact the same. Be as flexible as needed and let > > people do there crazy stuff, if they want to...* > > > > > > > > > > > > Looking at this level, priorities are IMO enough a feasible > > > > > > > mechanism, because it allows to easily add additional > > > > > > > layers transparently as needed ;) > > > While we still have a slightly different understanding about the 'why', > > we > > > DO agree about the result ;) > > > > > > But we really have to define which default ConfigSources we like to add > > > and which priority they should get. And we imo should not use 1, 2, > 3,.. > > > but like with @Priority for Interceptors we should start e.g. with 1000 > > and > > > keep big spaces in between them. Learned that with Basic in the early > > 80s... > > > > > *LOL, throwing me back at my old BASIC time. Save the queen,8,1 !!! Of > > course, completely agree!* > > > > > > > > Oh, for custom ConfigSources I actually don't care if the ConfigSource > > > itself knows it's ordinal or if the ordinal is set during registration > of > > > the ConfigSource. It is really very similar to the ELResolver chain > imo. > > > > *Yep. I would put on the provider side. So a PropertySourceProvider then > > would* > > *- register a top level property source* > > *- provide its priority needed to do the priorization* > > *- provide a property source as far it is avilable for the current > > environment* > > *- do whatever is needed internally (contextual management, classloader > > deps, caching etc). * > > * * > > *That way the programmatic combination case (the building SPI looks as a > > natural low level extension to this mechanism, if needed the same > mechanism > > could also be reapplied on level 2+).* > > > > > > > > I would not define priorities on single file level, but only on > > > property source level > > > > > > Actually in DeltaSpike each property file is an own ConfigSource. > > > > > *Often you define locations where files are to be located, such as > > META-INF/config/${STAGE}/**/*.xml . Bigger applications in Credit Suisse > > have dozens of such config files that are then combined to basically 2 > > different main config trees. So you dont't know the exact file names > > before. Nevertheless in such cases we handle inconsistencies within such > a > > property source as "Deployment Error" because there is no good > resolution > > algorithm there. Also given that the exact ordering how files are read is > > not relevant and the result will be reproducible.* > > > > > > > > Consider you have > > > > > > myframework.properties in the framework jar. Now I gonna use this jar > in > > > my application but like to slightly 'tweak' it's configuration. I just > > add > > > a myframework.properties to my own projects jar and give it a higher > > > ordinal (just add deltaspike_ordinal=1200 into the file). Voila. > > > > *I see here 2 use cases:* > > *1) the framework provides an explicit mechanism, how it enables being > > overriden. Configuration hereby is part of the framework API. This could > be > > additional files on the classpath, or a named-configuration provided by > > Tamaya, or ... (perhaps you have more ideas)... That is the way I would > say > > it should work in most cases.* > > > > *2) we have a case, where we want to explicitly override things that are > > not meant being overridden. Due to security reasons you might want to > > prevent that this is possible, so it might be a better idea to handle is > as > > a special case. One variant could be to define a specific > > PropertySourceFilter/Interceptor API, where we explicitly can override > > things, or we use the same priority overriding mechanism and add an > > additional PropertySourceProvider, with the same name, but higher > > priority.* > > > > *So assume we have "myframework-config" PropertySourceProvider with > prio > > 1000, we add a "myframework-config" PropertySourceProvider with prio > 1100. > > We could explicitly deny this feature for certain PropertySources, or > make > > it dependend on whatever conditions should be met. * > > > > *WDYT?* > > > > > > > > > > > > > > > > > Tamaya should be flexible enough to accomodate most of them, > > > > so application programmers still have their guides, where > > > > and how to add configuration files, resources etc. By adding > > > > > > > additional levels of configuration, e.g. on the deployment nodes, > > > > it is possible to override anything as needed. > > > With 'user' I was talking about the API. What you refer to is part of > the > > > SPI. And of course we will have this to add ConfigSources. > > > > > > > > > > > > > Summarizing I think we need both mechanisms. > > > > > > I still don't quite get this approach it seems. You have a default > > > configuration for the whole system which you get via > > > Configuration.current(), right? And you have a mechanism (builder like) > > to > > > tinker a different Configuration at runtime? Where do you store that? > > Where > > > do you use that? How to e.g. add graphical UI to allow admins to tweak > > > those settings? I still need a bit help to get the use case. What do > you > > > use it for usually? > > > > > *OK, hope that also some of the examples above help....* > > > > *Lets define a 2 level (for simplicity) source provider hierarchy:* > > > > - *"system" is provided globally once basically with a priority of > 1000 > > defined by PropertySourceProvider that registers it. Internally > > "system" is > > composed as follows (SPI code):* > > *private *PropertySource systemPS = > > PropertySourceBuilder.of("system") > > .addPaths("classpath:META-INF/cfg/**.xml", > > "classpath:META-INF/cfg/**.ini) > > > > .withAggregationPolicy(AggregationPolicy.IGNORE_DUPLICATES_AND_LOG) > > .addPaths("classpath:configs/*.conf") // legacy > location > > .addSystemProperties() > > .build(); > > - *"application" provides an isolated version, depending on the > > "applicationID" environment property. The corresponding > > **PropertySourceProvider > > **is added with a priority of 2000 for example and contains the > > following code (ignoring synchronization and cleanup for simplicity):* > > String applicationId = > > Environment.getInstance().get("applicationId"); > > if(applicationID==*null*) return *null*; > > PropertySource appPS = applicationConfigs.get(applicationID); > > if(appPS==*null*){ > > appPS = PropertySourceBuilder.of("application") > > .addPaths("classpath:META-INF/ > cfg/"+applicationID+"/*.xml", > > "classpath:META-INF/cfg/"+applicationID+"/*.ini") > > > > .withAggregationPolicy(AggregationPolicy.IGNORE_DUPLICATES_AND_LOG) > > .addPaths("classpath:configs/"+applicationID+"/*.conf") > // > > legacy location > > .withAggregationPolicy(CustomRequestBasedOverrides.of()) > > .addProvider(new > > MyApplicationRequestPropertySourceProvider()) > > .build(); > > } > > return appPS; > > > > *The user will not see any kind of these hierarchy, but simply access > > Configuration.current(). But for me the building of the internals of the > > provided building blocks is much more natural and flexible using the > > builder pattern...* > > > > *Now it's your turn again ;)* > > > > > > > > > > > > > > > > > LieGrue, > > > strub > > > > > > > > > > > > On Saturday, 27 December 2014, 18:47, Anatole Tresch < > [email protected] > > > > > > wrote: > > > > > > > >Hi Mark > > > > > > > > > > > >I will try to answer as precise as possible. I see a chance that we > are > > > aligning now our ideas more and more ;) > > > >... (inline) > > > > > > > > > > > >2014-12-27 16:26 GMT+01:00 Mark Struberg <[email protected]>: > > > > > > > >Hi Anatole! > > > >> > > > >>First of all thanks for sharing this sample. You might know your use > > > cases but others don't. > > > >>I assume that all this PropertySource construction is done at a > single > > > point, e.g. a producer metho > > > >> > > > >Yes, but that is only the access point, read ahead... > > > > > > > >Let me ask a few questions: > > > >> > > > >>1.) How do you know which ConfigSources exist in the project? You > might > > > be one jar in a dozen ones, having no control over them? > > > >> > > > >Basically you have to register PropertySourceProviderSpi's (or > whatever > > > they are named) with your code. How these are loaded depends on the > > > ServiceContext, so in SE you will probably use ServiceLoader only, > > whereas > > > in EE you can additionally rely on CDI to resolve corresponding > sources. > > > And as you said there could be multiple ones, which are visible in a > > > certain context depends on the ServiceContext. So for individual > property > > > sources (the ones != default) there is no issue I think, because > > > prioritization is not an issue, additionally individual configs can > also > > > use other existing ones in their metamodel, if needed (in the > definition > > > code of the provider itself). > > > > > > > > > > > >Level 1: globally the hierarchy of property sources, e.g. > > > > > > > > > > > >1. Boot Config, > > > >2. System config, > > > >3. Ear Config, > > > >4. App Config > > > >5. SaaS Config, > > > > > > > > > > > >I assumed that this hierarchy is well defined in an environment, and > > > typically not dynamic during runtime and therefore yes, hardcoded in > the > > > metamodel loaded on boot level. I even think from a security > perspective > > it > > > is not a good idea to have it dynamically. A provider may not be > > > available (based on the environment), so when executing a MDB in the > ear > > > it might be only the following sources are available/actively > providing > > > data to the current configuration: > > > > > > > > > > > >1. Boot Config, > > > >2. System config, > > > >3. Ear Config, > > > > > > > > > > > >Looking at this level, priorities are IMO enough a feasible mechanism, > > > because it allows to easily add additional layers transparently as > > needed ;) > > > > > > > > > > > >Now, what a Level 1 PropertySource is doing internally is completely > > > another story. Here in that area I expect more fine grained aggregation > > > operations to be used in many cases, as suggested in the example. > > > Additionally a PropertySourceProviderSpi is allowed to provide the > > > effective property sources contextually, e.g. depending on the current > > > classloader visiblity of items. > > > > > > > > > > > >So given that I adding a priority to a PropertySource is an > interesting > > > aspect. I even would ask, if we have one, if it would make sense to > > provide > > > some admin functionality to change the priority during runtime, or > adding > > > additional sources on the fly. When I am using the builder, priorities > > are > > > normally ignored, whereas for the overall configuration hierarchy, they > > are > > > useful. Additionally we also can use the priorities as well on Level 2 > > and > > > beyond, as useful ;) > > > > > > > > > > > >Summarizing I think we need both mechanisms. > > > > > > > > > > > > > > > > > > > >2.) In DeltaSpike each property file could define it's own ordinal > with > > a > > > simple entry in the property file. That way you can just add a JAR > which > > > overrides a default config from another jar. Would each of them get an > > own > > > name? Or do they all share the same name and priority? > > > >> > > > >I would not define priorities on single file level, but only on > > property > > > source level. I can still an additional property source and override > the > > > keys I want on source level. But as I said, do we really want > application > > > deploy an additional jar, hereby completely overriding any kind of > > config. > > > I would say, no. But we can define (company specific mechanism), how > and > > > when config can be overridden as part of the metamodel. > > > > > > > >3.) How do you deal with modularity? E.g. adding a dependency which > > needs > > > some other ConfigSource but knows nothing about your code? > > > >> > > > >>4.) Another modularity question: What about having multiple jars > > > (different independent framework parts) and each of them likes to have > > > another PropertySouceBuilder configuration? > > > >> > > > >Use different property source names on top level. > > > > > > > > > > > > > > > >>5.) How do you allow containers and framework programmers to add > > > PropertySources which affect all the project? > > > >>They define the metamodel and the extension points by the > > > implementation of the configuration metamodel. This metamodel as I > said > > > should not be dynamic. > > > >> > > > >>6.) > > > >> > > > >>> - you can override along the classloader hierarchy (system, ear, > war) > > > >> > > > >>> by adding resources to the classpath at xxx.Usually this does not > > work > > > well as e.g. all the JARs in WEB-INF/lib have exactly the same 'level'. > > > It's totally up to the ClassLoader mechanism in which order you get > them. > > > And that could change on every restart. Been there, done that - doesn't > > > work. > > > >> > > > > > > > >I would never rely on the classloader ordering. Normally your config > > > within such a level should be consistent. If not you have multiple > > > possibilities: > > > >1) just take one. > > > >2) Just take one and log. > > > >3) throw a Deployment Error > > > >4) take a default > > > >5) Override the conflicting value on the next level of defined > config, > > > so the conflict is not relevant anymore. How this is done, is defined > by > > > your metamodel. > > > > > > > > > > > > > > > >7.) > > > >>> In code the do simply: Configuration config = > > > >>> Configuration.current("Stephen"); > > > >>And what if another jar does Configuration.current("Giovanni")? > > > >>Imo that approach doesn't scale. > > > >> > > > >> > > > >Normally you do Configuration.current(); // the default > > > > > > > > > > > >Configuration.current("name") is for specific isolated/modular > > > configuration sources (see 3). > > > > > > > > > > > >>I like to align our ideas, so I enumerate my main goals and design > > > principles of a config system as I envision it: > > > >> > > > >>A.) have a total separation between the configuration system and it's > > > consumers. The 'consumers' API should not know where the result comes > > from > > > and also not what the value is. I think this is the only way we can > > provide > > > an extensible system where e.g. the Operations team could 'overwrite' > > > certain values. If we require the userland code to know about the > > > PropertySources then we cannot do this. > > > >> > > > > > > > > > > > >Partially agree. Companies have their own way how they configure > their > > > applications. Tamaya should be flexible enough to accomodate most of > > them, > > > so application programmers still have their guides, where and how to > add > > > configuration files, resources etc. By adding additional levels of > > > configuration, e.g. on the deployment nodes, > > > >it is possible to override anything as needed. The ladder, yes, should > > be > > > completely transparent. > > > > > > > > > > > >B.) Have a flexible SPI to allow Container Vendors and also > > application > > > programmers to add their own PropertySources in a very easy way. > > > >> > > > >Yes and no (see security constraints). I think we have to discuss > more > > > deeply the roles involved and what should be possible. > > > > > > > >C.) The SPI should be extensible via plugin mechanism. If a jar > > > containing a PropertySource gets added then it should automatically get > > > picked up for this very application. > > > >> > > > >If the metamodel foresees that to be so, yes. If not, not (see > > > secuirity...). > > > > > > > >Are we on the same page here? > > > >> > > > >> > > > > > > > > > > > >I hope! > > > > > > > > > > > >LieGrue, > > > >>strub > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >>> On Saturday, 27 December 2014, 15:35, Anatole Tresch < > > > [email protected]> wrote: > > > >>> > Hi Gerhard > > > >>> > > > >>> all you say is true here, I also will not provide every kind of > > > combination > > > >>> logic, but I want to have a mechanism that is capable enough. > > > >>> > > > >>> @Mark end users don't see this mechanisms here. It used by the > power > > > users, > > > >>> which define the "metamodel" how/where configuration is read > > > >>> and assembled, e.g. > > > >>> > > > >>> SPI: PropertySourceProvider { > > > >>> Collection<PropertySource> getPropertySources(); > > > >>> } > > > >>> > > > >>> Lets assume the provider is returning one property source named > > > >>> "Stephen". > > > >>> The power user would then probably write something like that: > > > >>> > > > >>> PropertySource src = > > > >>> PropertySourceBuilder.of("Stephen").addProviders(myProv2, > > > >>> myProv3).withAggregationPolicy(mySpecialPolicy) > > > >>> > > > >>> > > > > > .addProvider(BranchSpecificOverrideFilter.getPropertySource()). > withAggregationPolicy(AggregationPolicy.OVERRIDE_AND_LOG) > > > >>> .addProvider(DefaultJavaEEProviders.getEarProvider()).build(); > > > >>> > > > >>> This src instance then is returned e.g. by the provider (could also > > be > > > >>> contextually per classloader etc.). So it is not only that the > > > mechanism is > > > >>> much more explicit and powerful, it > > > >>> is also easier to understand. There is exactly one location, where > > all > > > >>> happens: the PropertySourceProvider. You look at the code and > > > immediately > > > >>> now, what is happening. > > > >>> So the code that is assembling the final property source returned > is > > > under > > > >>> full control of the users and nicely explicit. > > > >>> > > > >>> For normal users the mechanism is completely transparent the will > > > typically > > > >>> have some kind of documentation, where its written: > > > >>> > > > >>> - put your config files in the formats .ini or .xml at the location > > xy > > > >>> - you can override along the classloader hierarchy (system, ear, > war) > > > by > > > >>> adding resources to the classpath at xxx. > > > >>> - Branch sepcific overrides are merge automatically into it, see > > > chapter > > > >>> 3.4 for additional details. > > > >>> - Finally it is possible to override entries an system leven, by > > > setting > > > >>> according system properties. > > > >>> > > > >>> In code the do simply: Configuration config = > > > >>> Configuration.current("Stephen"); > > > >>> > > > >>> You get it? > > > >>> > > > >>> > > > >>> > > > >>> 2014-12-27 15:13 GMT+01:00 Gerhard Petracek < > > > [email protected]>: > > > >>> > > > >>>> hi anatole, > > > >>>> > > > >>>> i guess everybody agrees that overriding based on ordinals > wouldn't > > > be > > > >>>> enough for every possible use-case. > > > >>>> back then we had a long discussion about it. we ended up with > > adding > > > a > > > >>>> simple (and optional) ConfigFilter interface. > > > >>>> > > > >>>> @aggregation: > > > >>>> aggregation of key/value pairs is supported implicitly by an > > ordinal > > > based > > > >>>> approach. > > > >>>> aggregation of values >per key< can be done manually with an own > > > >>>> config-source (which delegates to the others and only aggregates > > > those > > > >>>> results). > > > >>>> yes - that way this use-case is more complicated out-of-the-box. > > > however, > > > >>>> imo if users (think they) need such a complicated config, they > > > should also > > > >>>> need to do a bit more for it. > > > >>>> (i guess in several cases they might question the requirement > > before > > > they > > > >>>> continue...) > > > >>>> > > > >>>> in general about possible use-cases (and features): > > > >>>> we can never address everything out-of-the-box. > > > >>>> a nice example is bean-validation. i could list a lot of cases > > which > > > >>> aren't > > > >>>> possible with that api/spi out-of-the-box, however, for the > > majority > > > >>> it's > > > >>>> good enough. > > > >>>> i saw several users who couldn't replace their existing > > > >>> validation-approach > > > >>>> (as it was). > > > >>>> however, several of them just simplified their validation-logic > and > > > were > > > >>>> still happy about the result. > > > >>>> others extended bv e.g. via the spi and only few kept what they > > used > > > before > > > >>>> (e.g. because it wasn't possible with bv or the workaround was > too > > > >>>> complicated > > > >>>> and they insisted on what they had). > > > >>>> > > > >>>> regards, > > > >>>> gerhard > > > >> > > > >>>> > > > >>>> > > > >>>> > > > >> > > > > > > > > > > > > > > > > > > > >-- > > > > > > > >Anatole Tresch > > > >Java Engineer & Architect, JSR Spec Lead > > > >Glärnischweg 10 > > > >CH - 8620 Wetzikon > > > > > > > > > > > >Switzerland, Europe Zurich, GMT+1 > > > >Twitter: @atsticks > > > >Blogs: http://javaremarkables.blogspot.ch/ > > > >Google: atsticks > > > >Mobile +41-76 344 62 79 > > > > > > > > > > > > > > > > > > > -- > > *Anatole Tresch* > > Java Engineer & Architect, JSR Spec Lead > > Glärnischweg 10 > > CH - 8620 Wetzikon > > > > *Switzerland, Europe Zurich, GMT+1* > > *Twitter: @atsticks* > > *Blogs: **http://javaremarkables.blogspot.ch/ > > <http://javaremarkables.blogspot.ch/>* > > > > *Google: atsticksMobile +41-76 344 62 79* > > >
