Configuration property sources with Sling (was: /etc/map with Placeholders for farms / dev stack)

[Robert Munteanu]

Hi,

On Tue, 2018-07-31 at 01:29 +0200, Georg Henzler wrote:
> A property file should still be possible... so if we do this
> in Sling, I think the following is needed:
> 
> Property Sources to be supported at minimum:
> 
> * An OSGi configuration
> * A local properties file
> * OS Env variables
> * Java System Properties

Bringing this out a new thread as I think it will slip by otherwise.

IIUC, you suggest that we extend the Sling configuration mechanism by
making other sources first-class besides OSGi. What are the use cases
for that? Currently we delegate almost everything to OSGi component
configurations, with some settings falling back or reading from
framework properties ( which can also fall back to system properties ).

Thanks,

Robert

Re: /etc/map with Placeholders for farms / dev stack

[Georg Henzler]

Hi Ruben,


... may not be the best solution as it would imply a package install
every time a change to the configs are necessary


Updating upon changes is an whole own challenge for your approach -
if resource change listeners only react on changes of placeholder
strings (e.g. 'mystr-${myVar}') because that's the only thing saved
in the JCR and replacement is done across various Sling modules,
there needs to be a new mechanism to listen to changes to property
changes (for the case the value of ${myVar} changes in the previous
example).


Not sure if maybe this would need a bigger spec to continue the work
and alignment on what everybody would like to see


I think this would be good. Maybe best to collect goals and non-goals
on a wiki page at [1]

-Georg


[1] https://cwiki.apache.org/confluence/display/SLING

Re: /etc/map with Placeholders for farms / dev stack

[Andreas Schaefer]

Hi George

> On Jul 30, 2018, at 4:29 PM, Georg Henzler  wrote:
> 
> Hi Andreas,
> 
>> Install Hooks have their own issues like that they do not work for
>> replication
> 
> Install Hooks work flawlessly with replication since around two years
> (AEM 6.3)

Cool. I probably ran into that a little bit before that.

> 
>>> ... allowing external URLs is dangerous...
>> I do agree with that and I will remove that from my POC.
> 
> A property file should still be possible... so if we do this
> in Sling, I think the following is needed:
> 
> Property Sources to be supported at minimum:
> 
> * An OSGi configuration
> * A local properties file
> * OS Env variables
> * Java System Properties

All but the local properties file I wanted to support. I cannot tell about the 
security implication of allowing a local properties file to avoid hackers to 
inject code.

> 
> There should be an Felix console plugin to show what is active
> from what source and where the value is used. It should live in
> a bundle named like e.g. org.apache.sling.systemenv and provide
> a simple interface to perform string interpolation on any
> string. This service can then be used by
> 
> * Resource resolver
> * Sling distribution
> * Context-Aware Configuration

The StringInterpolationProvider should not reside inside the Resource Resolver 
but in its own or a commons package. For my POC I just kept it there for 
convenience.

> 
> a product like AEM can also use this interface in
> 
> * AEM replication config
> * externaliser config
> 
> To have this mechanism properly rolled out it will take some
> time (until all listed modules properly use the to-defined
> interface)
> 
> IMHO it is not an option to do it locally in resourceresolver
> because

I started with the etc-mapping as this was the initial use case to move the 
ball forward.

> 
> a) it would result in quite a bit of duplicated code across the
> modules listed above (once we actually start implement it)
> b) if additional sources need to be added in the future (think
> ZooKeeper as one interesting option to receive those env-specific
> values), all consumers would have to be updated (unlikely to
> happen, more likely is to have inconsistent behaviour over time)

The String Interpolation should be not be in the Resource Resolver and hence 
can be easily used from other modules. Creating a separate module / git repo 
for just one class this seems to be a little bit of an overkill. Is there any 
place where we can place that?

> 
> Since the described above is quite a bit of effort, I opt for
> pushing a lower-level approach forward (something like [1] or
> maybe even something on oak level). Then there is only one
> implementation needed and it's available immediately, everywhere.

The ticket below is closed with ‘won’t fix’. So I am not sure if Jackrabbit 
wants to do that.

> 
> -Georg
> 
> [1] https://issues.apache.org/jira/browse/JCRVLT-254
> 
> 
> 

Re: /etc/map with Placeholders for farms / dev stack

[Ruben Reusser]

Georg,

I think this is a good discussion - looks like the sling team has to 
make some decisions here and I am glad the etc/map work Andreas did is 
at least sparking a bigger discussion.


From my point of view using packages and install hooks may work but may 
not be the best solution as it would imply a package install every time 
a change to the configs are necessary.


Not sure if maybe this would need a bigger spec to continue the work and 
alignment on what everybody would like to see


Ruben

On 7/30/2018 4:29 PM, Georg Henzler wrote:

Hi Andreas,


Install Hooks have their own issues like that they do not work for
replication


Install Hooks work flawlessly with replication since around two years
(AEM 6.3)


and they requires to changes whatever the customer has /
provides in order to add the place holders.


not sure what you mean with this?


... allowing external URLs is dangerous...

I do agree with that and I will remove that from my POC.


A property file should still be possible... so if we do this
in Sling, I think the following is needed:

Property Sources to be supported at minimum:

* An OSGi configuration
* A local properties file
* OS Env variables
* Java System Properties

There should be an Felix console plugin to show what is active
from what source and where the value is used. It should live in
a bundle named like e.g. org.apache.sling.systemenv and provide
a simple interface to perform string interpolation on any
string. This service can then be used by

* Resource resolver
* Sling distribution
* Context-Aware Configuration

a product like AEM can also use this interface in

* AEM replication config
* externaliser config

To have this mechanism properly rolled out it will take some
time (until all listed modules properly use the to-defined
interface)

IMHO it is not an option to do it locally in resourceresolver
because

a) it would result in quite a bit of duplicated code across the
modules listed above (once we actually start implement it)
b) if additional sources need to be added in the future (think
ZooKeeper as one interesting option to receive those env-specific
values), all consumers would have to be updated (unlikely to
happen, more likely is to have inconsistent behaviour over time)

Since the described above is quite a bit of effort, I opt for
pushing a lower-level approach forward (something like [1] or
maybe even something on oak level). Then there is only one
implementation needed and it's available immediately, everywhere.

-Georg

[1] https://issues.apache.org/jira/browse/JCRVLT-254

Re: /etc/map with Placeholders for farms / dev stack

[Georg Henzler]

Hi Andreas,


Install Hooks have their own issues like that they do not work for
replication


Install Hooks work flawlessly with replication since around two years
(AEM 6.3)


and they requires to changes whatever the customer has /
provides in order to add the place holders.


not sure what you mean with this?


... allowing external URLs is dangerous...

I do agree with that and I will remove that from my POC.


A property file should still be possible... so if we do this
in Sling, I think the following is needed:

Property Sources to be supported at minimum:

* An OSGi configuration
* A local properties file
* OS Env variables
* Java System Properties

There should be an Felix console plugin to show what is active
from what source and where the value is used. It should live in
a bundle named like e.g. org.apache.sling.systemenv and provide
a simple interface to perform string interpolation on any
string. This service can then be used by

* Resource resolver
* Sling distribution
* Context-Aware Configuration

a product like AEM can also use this interface in

* AEM replication config
* externaliser config

To have this mechanism properly rolled out it will take some
time (until all listed modules properly use the to-defined
interface)

IMHO it is not an option to do it locally in resourceresolver
because

a) it would result in quite a bit of duplicated code across the
modules listed above (once we actually start implement it)
b) if additional sources need to be added in the future (think
ZooKeeper as one interesting option to receive those env-specific
values), all consumers would have to be updated (unlikely to
happen, more likely is to have inconsistent behaviour over time)

Since the described above is quite a bit of effort, I opt for
pushing a lower-level approach forward (something like [1] or
maybe even something on oak level). Then there is only one
implementation needed and it's available immediately, everywhere.

-Georg

[1] https://issues.apache.org/jira/browse/JCRVLT-254

Re: /etc/map with Placeholders for farms / dev stack

[Andreas Schaefer]

Hi George

> On Jul 26, 2018, at 2:21 PM, Georg Henzler  wrote:
> 
> Hi Andreas,
> 
> I'm not sure if this is going in the right direction - this is currently
> implemented at at a local module level 
> (sling-org-apache-sling-resourceresolver)
> when it is really something that the platform should support (see below).

Ruben and I were discussing to add placeholders also to OSGi Configurations to 
be able to change host names for both the etc-mapping but also for Distribution 
/ Replication.
That would requires to add the string interpolation to Felix in order to make 
it work as just trying to handle the OSG configuration from JCR leads to many 
issues.
But for now we would love to see this in Sling as soon as possible as testing 
of bigger companies is difficult and time consuming.
If we have a solution in Jackrabbit I am more than happy to remove my code from 
the resource resolver.

> 
> Also note that your example/reasoning in the description of SLING-7768
> only lists DEV/QA/STAGING/PROD - if you are limited to those environments you
> can easily do with runmodes. Placeholder replacement becomes interesting when
> you think about an "elastic number of environments" that are not known at
> development time (think about spinning up "STAGING-7" for that special 
> performance
> test that needs to run in isolation)

This was the basic test case but our case this involves to have a different 
host name because of a temporary migration server.
It would also work for URL paths as well.

> 
>> ... Ruben and I think it could be helpful to make the Placeholder Provider
>> more generic to that it could be used in other services like the
>> distribution/replication, externalizers etc to have an easier way to
>> manage server configuration in a central place.
> 
> I agree we should have the capability in the platform, but IMHO it should not
> be in sling-org-apache-sling-resourceresolver. There was a discussion in
> jackrabbit-dev:
> 
> http://jackrabbit.510166.n4.nabble.com/Environment-specific-non-role-based-configurations-td4669795.html
> and https://issues.apache.org/jira/browse/JCRVLT-254
> for the initial issue.
> 
> Btw.: you can achieve your requirement already today (down to AEM 6.0) with
> https://github.com/Netcentric/system-env-install-hook
> (but I'd rather see this as first-class citizen as part of the platform
> and not as a third-party module)

Install Hooks have their own issues like that they do not work for replication 
and they requires to changes whatever the customer has / provides in order to 
add the place holders.

> 
>>> * No need to support anything else like files and system properties to
>>> provide the placeholders. Makes error handling much more complicated (what
>>> do you do if the http download of that placeholder file fails?), and I
>>> would rather leave it to the ops team to implement that process and handle
>>> any errors.
>> I see the issue with the external files / web resources. The system
>> properties are there to make sure an admin can override any settings
>> if things do not work out.
> 
> I agree here with Jörg that allowing external URLs is dangerous here (and
> it could even be seen critical from a security perspective). Also, the config 
> allows
> to give both key value pairs directly and a URL to retrieve them from - I 
> suppose
> these two property sets are merged?

I do agree with that and I will remove that from my POC.

> 
> But to sum up I think the best would be to have a mechanism for placeholder 
> replacement
> on a lower level...
> 
> -Georg

Re: /etc/map with Placeholders for farms / dev stack

[Georg Henzler]

Hi Andreas,

I'm not sure if this is going in the right direction - this is currently
implemented at at a local module level 
(sling-org-apache-sling-resourceresolver)
when it is really something that the platform should support (see 
below).


Also note that your example/reasoning in the description of SLING-7768
only lists DEV/QA/STAGING/PROD - if you are limited to those 
environments you
can easily do with runmodes. Placeholder replacement becomes interesting 
when
you think about an "elastic number of environments" that are not known 
at
development time (think about spinning up "STAGING-7" for that special 
performance

test that needs to run in isolation)

... Ruben and I think it could be helpful to make the Placeholder 
Provider

more generic to that it could be used in other services like the
distribution/replication, externalizers etc to have an easier way to
manage server configuration in a central place.


I agree we should have the capability in the platform, but IMHO it 
should not

be in sling-org-apache-sling-resourceresolver. There was a discussion in
jackrabbit-dev:

http://jackrabbit.510166.n4.nabble.com/Environment-specific-non-role-based-configurations-td4669795.html
and https://issues.apache.org/jira/browse/JCRVLT-254
for the initial issue.

Btw.: you can achieve your requirement already today (down to AEM 6.0) 
with

https://github.com/Netcentric/system-env-install-hook
(but I'd rather see this as first-class citizen as part of the platform
and not as a third-party module)


* No need to support anything else like files and system properties to
provide the placeholders. Makes error handling much more complicated 
(what

do you do if the http download of that placeholder file fails?), and I
would rather leave it to the ops team to implement that process and 
handle

any errors.


I see the issue with the external files / web resources. The system
properties are there to make sure an admin can override any settings
if things do not work out.


I agree here with Jörg that allowing external URLs is dangerous here 
(and
it could even be seen critical from a security perspective). Also, the 
config allows
to give both key value pairs directly and a URL to retrieve them from - 
I suppose

these two property sets are merged?

But to sum up I think the best would be to have a mechanism for 
placeholder replacement

on a lower level...

-Georg

Re: /etc/map with Placeholders for farms / dev stack

[Bertrand Delacretaz]

Hi,

On Thu, Jul 26, 2018 at 12:51 AM Andreas Schaefer
 wrote:
> ...I am wondering if Sling has some Integration Tests (IT). I could not find 
> any further tests regarding /etc/map. ..

Our "global" integration tests are at
https://github.com/apache/sling-org-apache-sling-launchpad-integration-tests
and https://github.com/apache/sling-org-apache-sling-launchpad-test-services
but over time we've been trying to bring those closer to the code that
they test, so many modules have ITs directly in their codebase.

I suspect we might be lacking such tests for /etc/map, there's very
few test files there which have "map" in their name, and no files
under src which contain the string "etc/map".

-Bertrand

Re: /etc/map with Placeholders for farms / dev stack

[Carsten Ziegeler]

Hi,

I only had a brief look at the current code, and currently it is
exporting the package org.apache.sling.resourceresolver.impl.mapping
which looks wrong to me. If we need new API this should go into the API
bundle. But I would prefer if we can get away without a new interface

Carsten


Andreas Schaefer wrote
> The placeholder was renamed to sling interpolation, changed the delimiter to 
> ${} and added a jUnit test suite where the StringInterpolationProvider 
> replacement is tested through Map Entries. 
> 
> I am wondering if Sling has some Integration Tests (IT). I could not find any 
> further tests regarding /etc/map.
> 
> Cheers - Andreas Schaefer
> 
>> On Jul 12, 2018, at 1:15 AM, Bertrand Delacretaz  
>> wrote:
>>
>> Hi,
>>
>> On Mon, Jul 9, 2018 at 4:18 PM Andreas Schaefer
>>  wrote:
>>> ...Ruben and I think it could be helpful to make the Placeholder Provider 
>>> more generic...
>>
>> I agree with that and I think "string interpolation" is a better name
>> than "placeholder" which is quite generic - as per
>> https://en.wikipedia.org/wiki/String_interpolation
>>
>> -Bertrand
> 
-- 
Carsten Ziegeler
Adobe Research Switzerland
cziege...@apache.org

Re: /etc/map with Placeholders for farms / dev stack

[Andreas Schaefer]

The placeholder was renamed to sling interpolation, changed the delimiter to 
${} and added a jUnit test suite where the StringInterpolationProvider 
replacement is tested through Map Entries. 

I am wondering if Sling has some Integration Tests (IT). I could not find any 
further tests regarding /etc/map.

Cheers - Andreas Schaefer

> On Jul 12, 2018, at 1:15 AM, Bertrand Delacretaz  
> wrote:
> 
> Hi,
> 
> On Mon, Jul 9, 2018 at 4:18 PM Andreas Schaefer
>  wrote:
>> ...Ruben and I think it could be helpful to make the Placeholder Provider 
>> more generic...
> 
> I agree with that and I think "string interpolation" is a better name
> than "placeholder" which is quite generic - as per
> https://en.wikipedia.org/wiki/String_interpolation
> 
> -Bertrand

Re: /etc/map with Placeholders for farms / dev stack

[Bertrand Delacretaz]

Hi,

On Mon, Jul 9, 2018 at 4:18 PM Andreas Schaefer
 wrote:
> ...Ruben and I think it could be helpful to make the Placeholder Provider 
> more generic...

I agree with that and I think "string interpolation" is a better name
than "placeholder" which is quite generic - as per
https://en.wikipedia.org/wiki/String_interpolation

-Bertrand

Re: /etc/map with Placeholders for farms / dev stack

[Andreas Schaefer]

> On Jul 7, 2018, at 4:26 AM, Jörg Hoh  wrote:
> 
> Hi Andreas,
> 
> 2018-07-06 20:59 GMT+00:00 Andreas Schaefer :
> 
>> 
>> 
>> https://issues.apache.org/jira/browse/SLING-7768 <
>> https://issues.apache.org/jira/browse/SLING-7768>
>> 
> 
> 
> In this ticket I see that you allow not only the replacement of property
> values, but also of resource names. Is that required? IIRC you can also
> work with "sling:match" properties, so the name of the resource is ignored.
> You haven't posted any implementation yet, so I wonder if that limitation
> could reduce the amount of changes to actually implement it.

I just posted a branch to the org-apache-sling-resourceresolver module called 
feature/SLING-7768.

I think it would be best to support both ways (the node name as well as the 
sling:match) to make it easier on the user.
Ruben and I think it could be helpful to make the Placeholder Provider more 
generic to that it could be used in other services like the 
distribution/replication, externalizers etc to have an easier way to manage 
server configuration in a central place.

> 
> 
>> 
>> The OSGi Web Plugin for the Resource Resolver actually shows the resolved
>> values making it easy to understand and check the /etc/map entries.
>> 
>> For the Placeholder provider I also think it would be great if the
>> placeholder values could be provided from a URL (file or web page) and also
>> as CLI arguments:
>> 
>> java -jar sling*.jar -Dphv.fq.host.name=my-host.com 
>> 
>> This would be helpful if a user messed up the /etc/map and cannot get back
>> into Sling.
>> 
> 
> Nice feature. My whishlist for it:
> 
> * When I provide the placeholders via a file, this file should be monitored
> and reloaded as I change it. If I add more /etc/map entries during runtime
> and need to introduce new placeholders, I would like to update the
> placeholder file in parallel, and have the new entries effective without
> restarting.
> * No need to support anything else like files and system properties to
> provide the placeholders. Makes error handling much more complicated (what
> do you do if the http download of that placeholder file fails?), and I
> would rather leave it to the ops team to implement that process and handle
> any errors.

I see the issue with the external files / web resources. The system properties 
are there to make sure an admin can override any settings if things do not work 
out.

> * Good test coverage (the resource resolver grew into a complex beast over
> the years…)

Sure, will do.

> * I would prefer the "${placeholder}" format instead of "{{{placeholder}}}"
> format :-)

Do JCR node names support ${}? If so that can be easily changed.

> 
> 
> -- 
> Cheers,
> Jörg Hoh,
> 
> http://cqdump.wordpress.com
> Twitter: @joerghoh

Re: /etc/map with Placeholders for farms / dev stack

[Jörg Hoh]

Hi Andreas,

2018-07-06 20:59 GMT+00:00 Andreas Schaefer :

>
>
> https://issues.apache.org/jira/browse/SLING-7768 <
> https://issues.apache.org/jira/browse/SLING-7768>
>


In this ticket I see that you allow not only the replacement of property
values, but also of resource names. Is that required? IIRC you can also
work with "sling:match" properties, so the name of the resource is ignored.
You haven't posted any implementation yet, so I wonder if that limitation
could reduce the amount of changes to actually implement it.


>
> The OSGi Web Plugin for the Resource Resolver actually shows the resolved
> values making it easy to understand and check the /etc/map entries.
>
> For the Placeholder provider I also think it would be great if the
> placeholder values could be provided from a URL (file or web page) and also
> as CLI arguments:
>
> java -jar sling*.jar -Dphv.fq.host.name=my-host.com 
>
> This would be helpful if a user messed up the /etc/map and cannot get back
> into Sling.
>

Nice feature. My whishlist for it:

* When I provide the placeholders via a file, this file should be monitored
and reloaded as I change it. If I add more /etc/map entries during runtime
and need to introduce new placeholders, I would like to update the
placeholder file in parallel, and have the new entries effective without
restarting.
* No need to support anything else like files and system properties to
provide the placeholders. Makes error handling much more complicated (what
do you do if the http download of that placeholder file fails?), and I
would rather leave it to the ops team to implement that process and handle
any errors.
* Good test coverage (the resource resolver grew into a complex beast over
the years...)
* I would prefer the "${placeholder}" format instead of "{{{placeholder}}}"
format :-)


-- 
Cheers,
Jörg Hoh,

http://cqdump.wordpress.com
Twitter: @joerghoh

/etc/map with Placeholders for farms / dev stack

[Andreas Schaefer]

Hi

I create a ticket asking for the addition of placeholder support in /etc/map:

https://issues.apache.org/jira/browse/SLING-7768 


The OSGi Web Plugin for the Resource Resolver actually shows the resolved 
values making it easy to understand and check the /etc/map entries.

For the Placeholder provider I also think it would be great if the placeholder 
values could be provided from a URL (file or web page) and also as CLI 
arguments:

java -jar sling*.jar -Dphv.fq.host.name=my-host.com 

This would be helpful if a user messed up the /etc/map and cannot get back into 
Sling.

Let me know what you think.

Cheers - Andreas Schaefer