Re: Dependency Management

[Bertrand Delacretaz]

Hi,

On Fri, Jun 3, 2016 at 11:20 AM, Oliver Lietz  wrote:
> ...AFAIR we are moving ITs closer to the code under test (Bertrand?)...

Yes, I've been doing that and encouraging people to do so.

OTOH the existing integration test suite in
launchpad/integration-tests works and there's no urgent need to
reorganize it IMO.

-Bertrand

Re: Dependency Management

[Bertrand Delacretaz]

Hi,

On Tue, Jun 7, 2016 at 1:16 PM, Georg Henzler  wrote:
> ... just tried to run the latest with hc-core [1] to test something
> completely else and obviously it does not work with AEM anymore unless you
> deploy commons lang3 elsewhere...

I agree that that's bad, because you can replace "with AEM" by "in a
system where it used t work before" and I don't think there's a solid
reason for that lack of backwards compatibility.

Discussing such things in general terms can be hard, but if people see
concrete cases like this please create jira tickets for those specific
cases, and we can look at them and make good "local" decisions.

-Bertrand

> [1]
> https://github.com/apache/sling/commit/7203dc7ec1b7b85ecca93450179bbefa78f7bd02?diff=unified

Re: Dependency Management

[Georg Henzler]

On 2016-05-26 14:36, Bertrand Delacretaz wrote:

My opinion, and I think this is what we've been doing in Sling forever
even if we didn't explicitly document it, is that if a bundle B
depends on another *for its API* then the dependency should list the
lowest possible version.


I agree to this.

... just tried to run the latest with hc-core [1] to test something 
completely else and obviously it does not work with AEM anymore unless 
you deploy commons lang3 elsewhere - IMHO there is no need to make 
everyone go through this hassle (for me it was easier to revert the 
commit to test what I was testing...)


Regards
Georg

[1]
https://github.com/apache/sling/commit/7203dc7ec1b7b85ecca93450179bbefa78f7bd02?diff=unified

Re: Dependency Management

[Konrad Windszus]

> On 07 Jun 2016, at 10:32, Konrad Windszus  wrote:
> 
> Hi,
>> On 07 Jun 2016, at 10:12, Oliver Lietz  wrote:
>> 
>> On Monday 06 June 2016 14:57:16 Konrad Windszus wrote:
>>> I would like to wrap things up here:
>>> 
>>> So it seems the common agreement is:
>> 
>> Really?
>> 
>>> 1) Depend on the lowest possible version of a dependency (to let bnd
>>> generate import package ranges which are as broad as possible to support as
>>> many different systems as possible (even old ones)). A module should only
>>> be upgraded to a newer dependency in case this is really necessary (because
>>> of a feature being used which is not available in earlier version or if the
>>> newer version contains a bugfix from which the consumer bundle is
>>> affected).
>> 
>> I don't see all topics discussed and your definition is very vague. It can 
>> be 
>> taken as "never upgrade". This prevents modernization of our code base 
>> (example given).
> I agree that this is rather vague. We definitely need an official way of 
> upgrading dependencies. But in the cases of upgrading commons-lang and 
> commons-io I don't see any convincing reason to upgrade to the newest 
> dependency for the modules validation or health check (we don't actually 
> leverage new API there yet). Only if we do, we can and should discuss again.
> 
>> 
>>> 2) For ITs and the Maven Launchpad depend on the same version
>>> which is reasonably new (this is only possible if the IT is in a dedicated
>>> Maven bundle).
>> 
>> Pax Exam can pull in any version of a dependency even when running in the 
>> same 
>> module. The limitation to one version is only true for our current setup of 
>> unit tests with Maven 3.
> Not all ITs are based on Pax Exam. So the ones leveraging a real Sling 
> instance and being located in the same bundle as the code itself (Transporter 
> rule) are not that easy to decouple.
>> 
>>> 3) Since Maven does not completely separate compile and test
>>> classpath for the unit test we must depend on the lowest possible version
>>> as well.
>>> 
>>> If everyone is fine with that, I would ask Oliver Lietz to basically revert
>>> some changes being done in https://issues.apache.org/jira/browse/SLING-5603 
>>> 
>>> and https://issues.apache.org/jira/browse/SLING-5685 
>>> . Thanks for your
>>> input,
>> 
>> To make clear you are the driving force behind this approach please go ahead 
>> and revert yourself (and don't forget to adjust fixed versions).
> Ok, I will do so for Healthcheck Core in 
> https://issues.apache.org/jira/browse/SLING-5685 
> .
Actually it is unclear why the dependency to Commons IO was introduced at all 
for HC Core. Please also check 
https://issues.apache.org/jira/browse/SLING-5685?focusedCommentId=15318210&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15318210.
@Oli: Can you quickly comment on that? Thanks

>> 
>> Regards,
>> O.
>> 
>>> Konrad
>>> 
 On 03 Jun 2016, at 13:32, Oliver Lietz  wrote:
 
 On Friday 03 June 2016 20:13:29 Steven Walters wrote:
> On Fri, Jun 3, 2016 at 6:20 PM, Oliver Lietz  
>> wrote:
>> On Monday 30 May 2016 15:28:04 Robert Munteanu wrote:
>>> On Fri, 2016-05-27 at 11:14 +0200, Oliver Lietz wrote:
 I don't think there is support in Maven 3 for unit tests using a
 different
 version than for compile. Not sure if other build tools can handle
 different
 versions of dependencies for compile and test.
>>> 
>>> Maven 3 is unable to do that. I can't say anything for other build
>>> systems.
>> 
>> Looks like Gradle can do (but I haven't tested):
>> https://docs.gradle.org/current/userguide/java_plugin.html#sec:java_plug
>> in
>> _and_dependency_management
> 
> Yes, Gradle can do this.
> 'compile' and 'testRuntime' configurations are the standard
> configuration names that are relevant here and there is a natural
> inheritance chain (such that the default is that 'testRuntime'
> eventually inherits everything in 'compile'),
> but the versions used by 'testCompile/'testRuntime' can be overridden
> to be different (usually newer) ones than what is used by
> 'compile'/'runtime'
> 
> I've done this plenty in the past when working with AEM to use newer
> APIs/frameworks in the tests than what the code is compiled against.
> such as compiling for AEM 5.6, but due to using newer versions of the
> Sling mock frameworks in test cases, having newer Sling jars pulled in
> as a result for test case compilation + execution.
> Maven can't do this, so it makes writing tests for the same code more
> difficult when using Maven when compared to Gradle under these
> circumstances.
> 
> However, I don't particularly see this being enough reason alone to
> warrant 

Re: Dependency Management

[Konrad Windszus]

Hi,
> On 07 Jun 2016, at 10:12, Oliver Lietz  wrote:
> 
> On Monday 06 June 2016 14:57:16 Konrad Windszus wrote:
>> I would like to wrap things up here:
>> 
>> So it seems the common agreement is:
> 
> Really?
> 
>> 1) Depend on the lowest possible version of a dependency (to let bnd
>> generate import package ranges which are as broad as possible to support as
>> many different systems as possible (even old ones)). A module should only
>> be upgraded to a newer dependency in case this is really necessary (because
>> of a feature being used which is not available in earlier version or if the
>> newer version contains a bugfix from which the consumer bundle is
>> affected).
> 
> I don't see all topics discussed and your definition is very vague. It can be 
> taken as "never upgrade". This prevents modernization of our code base 
> (example given).
I agree that this is rather vague. We definitely need an official way of 
upgrading dependencies. But in the cases of upgrading commons-lang and 
commons-io I don't see any convincing reason to upgrade to the newest 
dependency for the modules validation or health check (we don't actually 
leverage new API there yet). Only if we do, we can and should discuss again.

> 
>> 2) For ITs and the Maven Launchpad depend on the same version
>> which is reasonably new (this is only possible if the IT is in a dedicated
>> Maven bundle).
> 
> Pax Exam can pull in any version of a dependency even when running in the 
> same 
> module. The limitation to one version is only true for our current setup of 
> unit tests with Maven 3.
Not all ITs are based on Pax Exam. So the ones leveraging a real Sling instance 
and being located in the same bundle as the code itself (Transporter rule) are 
not that easy to decouple.
> 
>> 3) Since Maven does not completely separate compile and test
>> classpath for the unit test we must depend on the lowest possible version
>> as well.
>> 
>> If everyone is fine with that, I would ask Oliver Lietz to basically revert
>> some changes being done in https://issues.apache.org/jira/browse/SLING-5603
>> and https://issues.apache.org/jira/browse/SLING-5685. Thanks for your
>> input,
> 
> To make clear you are the driving force behind this approach please go ahead 
> and revert yourself (and don't forget to adjust fixed versions).
Ok, I will do so for Healthcheck Core in 
https://issues.apache.org/jira/browse/SLING-5685.
> 
> Regards,
> O.
> 
>> Konrad
>> 
>>> On 03 Jun 2016, at 13:32, Oliver Lietz  wrote:
>>> 
>>> On Friday 03 June 2016 20:13:29 Steven Walters wrote:
 On Fri, Jun 3, 2016 at 6:20 PM, Oliver Lietz  
> wrote:
> On Monday 30 May 2016 15:28:04 Robert Munteanu wrote:
>> On Fri, 2016-05-27 at 11:14 +0200, Oliver Lietz wrote:
>>> I don't think there is support in Maven 3 for unit tests using a
>>> different
>>> version than for compile. Not sure if other build tools can handle
>>> different
>>> versions of dependencies for compile and test.
>> 
>> Maven 3 is unable to do that. I can't say anything for other build
>> systems.
> 
> Looks like Gradle can do (but I haven't tested):
> https://docs.gradle.org/current/userguide/java_plugin.html#sec:java_plug
> in
> _and_dependency_management
 
 Yes, Gradle can do this.
 'compile' and 'testRuntime' configurations are the standard
 configuration names that are relevant here and there is a natural
 inheritance chain (such that the default is that 'testRuntime'
 eventually inherits everything in 'compile'),
 but the versions used by 'testCompile/'testRuntime' can be overridden
 to be different (usually newer) ones than what is used by
 'compile'/'runtime'
 
 I've done this plenty in the past when working with AEM to use newer
 APIs/frameworks in the tests than what the code is compiled against.
 such as compiling for AEM 5.6, but due to using newer versions of the
 Sling mock frameworks in test cases, having newer Sling jars pulled in
 as a result for test case compilation + execution.
 Maven can't do this, so it makes writing tests for the same code more
 difficult when using Maven when compared to Gradle under these
 circumstances.
 
 However, I don't particularly see this being enough reason alone to
 warrant the effort of having Sling switching to Gradle.
 There are some other possible benefits to using Gradle, such as being
 able to merge separate IT modules back into modules they are testing,
 therefore reducing the number of modules/build complexity.
 But even so, I still don't see Sling leaving Maven.
>>> 
>>> Same here. That would be a lot of work and I don't think the project can
>>> perform such a big task right now.
>>> 
>>> O.
> 

Re: Dependency Management

[Oliver Lietz]

On Monday 06 June 2016 14:57:16 Konrad Windszus wrote:
> I would like to wrap things up here:
> 
> So it seems the common agreement is:

Really?

> 1) Depend on the lowest possible version of a dependency (to let bnd
> generate import package ranges which are as broad as possible to support as
> many different systems as possible (even old ones)). A module should only
> be upgraded to a newer dependency in case this is really necessary (because
> of a feature being used which is not available in earlier version or if the
> newer version contains a bugfix from which the consumer bundle is
> affected).

I don't see all topics discussed and your definition is very vague. It can be 
taken as "never upgrade". This prevents modernization of our code base 
(example given).

> 2) For ITs and the Maven Launchpad depend on the same version
> which is reasonably new (this is only possible if the IT is in a dedicated
> Maven bundle).

Pax Exam can pull in any version of a dependency even when running in the same 
module. The limitation to one version is only true for our current setup of 
unit tests with Maven 3.

> 3) Since Maven does not completely separate compile and test
> classpath for the unit test we must depend on the lowest possible version
> as well.
> 
> If everyone is fine with that, I would ask Oliver Lietz to basically revert
> some changes being done in https://issues.apache.org/jira/browse/SLING-5603
> and https://issues.apache.org/jira/browse/SLING-5685. Thanks for your
> input,

To make clear you are the driving force behind this approach please go ahead 
and revert yourself (and don't forget to adjust fixed versions).

Regards,
O.

> Konrad
> 
> > On 03 Jun 2016, at 13:32, Oliver Lietz  wrote:
> > 
> > On Friday 03 June 2016 20:13:29 Steven Walters wrote:
> >> On Fri, Jun 3, 2016 at 6:20 PM, Oliver Lietz  
wrote:
> >>> On Monday 30 May 2016 15:28:04 Robert Munteanu wrote:
>  On Fri, 2016-05-27 at 11:14 +0200, Oliver Lietz wrote:
> > I don't think there is support in Maven 3 for unit tests using a
> > different
> > version than for compile. Not sure if other build tools can handle
> > different
> > versions of dependencies for compile and test.
>  
>  Maven 3 is unable to do that. I can't say anything for other build
>  systems.
> >>> 
> >>> Looks like Gradle can do (but I haven't tested):
> >>> https://docs.gradle.org/current/userguide/java_plugin.html#sec:java_plug
> >>> in
> >>> _and_dependency_management
> >> 
> >> Yes, Gradle can do this.
> >> 'compile' and 'testRuntime' configurations are the standard
> >> configuration names that are relevant here and there is a natural
> >> inheritance chain (such that the default is that 'testRuntime'
> >> eventually inherits everything in 'compile'),
> >> but the versions used by 'testCompile/'testRuntime' can be overridden
> >> to be different (usually newer) ones than what is used by
> >> 'compile'/'runtime'
> >> 
> >> I've done this plenty in the past when working with AEM to use newer
> >> APIs/frameworks in the tests than what the code is compiled against.
> >> such as compiling for AEM 5.6, but due to using newer versions of the
> >> Sling mock frameworks in test cases, having newer Sling jars pulled in
> >> as a result for test case compilation + execution.
> >> Maven can't do this, so it makes writing tests for the same code more
> >> difficult when using Maven when compared to Gradle under these
> >> circumstances.
> >> 
> >> However, I don't particularly see this being enough reason alone to
> >> warrant the effort of having Sling switching to Gradle.
> >> There are some other possible benefits to using Gradle, such as being
> >> able to merge separate IT modules back into modules they are testing,
> >> therefore reducing the number of modules/build complexity.
> >> But even so, I still don't see Sling leaving Maven.
> > 
> > Same here. That would be a lot of work and I don't think the project can
> > perform such a big task right now.
> > 
> > O.

Re: Dependency Management

[Konrad Windszus]

I would like to wrap things up here:

So it seems the common agreement is:

1) Depend on the lowest possible version of a dependency (to let bnd generate 
import package ranges which are as broad as possible to support as many 
different systems as possible (even old ones)). 
A module should only be upgraded to a newer dependency in case this is really 
necessary (because of a feature being used which is not available in earlier 
version or if the newer version contains a bugfix from which the consumer 
bundle is affected).
2) For ITs and the Maven Launchpad depend on the same version which is 
reasonably new (this is only possible if the IT is in a dedicated Maven bundle).
3) Since Maven does not completely separate compile and test classpath for the 
unit test we must depend on the lowest possible version as well.

If everyone is fine with that, I would ask Oliver Lietz to basically revert 
some changes being done in https://issues.apache.org/jira/browse/SLING-5603 and 
https://issues.apache.org/jira/browse/SLING-5685.
Thanks for your input,
Konrad

> On 03 Jun 2016, at 13:32, Oliver Lietz  wrote:
> 
> On Friday 03 June 2016 20:13:29 Steven Walters wrote:
>> On Fri, Jun 3, 2016 at 6:20 PM, Oliver Lietz  wrote:
>>> On Monday 30 May 2016 15:28:04 Robert Munteanu wrote:
 On Fri, 2016-05-27 at 11:14 +0200, Oliver Lietz wrote:
> I don't think there is support in Maven 3 for unit tests using a
> different
> version than for compile. Not sure if other build tools can handle
> different
> versions of dependencies for compile and test.
 
 Maven 3 is unable to do that. I can't say anything for other build
 systems.
>>> 
>>> Looks like Gradle can do (but I haven't tested):
>>> https://docs.gradle.org/current/userguide/java_plugin.html#sec:java_plugin
>>> _and_dependency_management
>> Yes, Gradle can do this.
>> 'compile' and 'testRuntime' configurations are the standard
>> configuration names that are relevant here and there is a natural
>> inheritance chain (such that the default is that 'testRuntime'
>> eventually inherits everything in 'compile'),
>> but the versions used by 'testCompile/'testRuntime' can be overridden
>> to be different (usually newer) ones than what is used by
>> 'compile'/'runtime'
>> 
>> I've done this plenty in the past when working with AEM to use newer
>> APIs/frameworks in the tests than what the code is compiled against.
>> such as compiling for AEM 5.6, but due to using newer versions of the
>> Sling mock frameworks in test cases, having newer Sling jars pulled in
>> as a result for test case compilation + execution.
>> Maven can't do this, so it makes writing tests for the same code more
>> difficult when using Maven when compared to Gradle under these
>> circumstances.
>> 
>> However, I don't particularly see this being enough reason alone to
>> warrant the effort of having Sling switching to Gradle.
>> There are some other possible benefits to using Gradle, such as being
>> able to merge separate IT modules back into modules they are testing,
>> therefore reducing the number of modules/build complexity.
>> But even so, I still don't see Sling leaving Maven.
> 
> Same here. That would be a lot of work and I don't think the project can 
> perform such a big task right now.
> 
> O.
> 

Re: Dependency Management

[Oliver Lietz]

On Friday 03 June 2016 20:13:29 Steven Walters wrote:
> On Fri, Jun 3, 2016 at 6:20 PM, Oliver Lietz  wrote:
> > On Monday 30 May 2016 15:28:04 Robert Munteanu wrote:
> > > On Fri, 2016-05-27 at 11:14 +0200, Oliver Lietz wrote:
> > > > I don't think there is support in Maven 3 for unit tests using a
> > > > different
> > > > version than for compile. Not sure if other build tools can handle
> > > > different
> > > > versions of dependencies for compile and test.
> > > 
> > > Maven 3 is unable to do that. I can't say anything for other build
> > > systems.
> > 
> > Looks like Gradle can do (but I haven't tested):
> > https://docs.gradle.org/current/userguide/java_plugin.html#sec:java_plugin
> > _and_dependency_management
> Yes, Gradle can do this.
> 'compile' and 'testRuntime' configurations are the standard
> configuration names that are relevant here and there is a natural
> inheritance chain (such that the default is that 'testRuntime'
> eventually inherits everything in 'compile'),
> but the versions used by 'testCompile/'testRuntime' can be overridden
> to be different (usually newer) ones than what is used by
> 'compile'/'runtime'
> 
> I've done this plenty in the past when working with AEM to use newer
> APIs/frameworks in the tests than what the code is compiled against.
> such as compiling for AEM 5.6, but due to using newer versions of the
> Sling mock frameworks in test cases, having newer Sling jars pulled in
> as a result for test case compilation + execution.
> Maven can't do this, so it makes writing tests for the same code more
> difficult when using Maven when compared to Gradle under these
> circumstances.
> 
> However, I don't particularly see this being enough reason alone to
> warrant the effort of having Sling switching to Gradle.
> There are some other possible benefits to using Gradle, such as being
> able to merge separate IT modules back into modules they are testing,
> therefore reducing the number of modules/build complexity.
> But even so, I still don't see Sling leaving Maven.

Same here. That would be a lot of work and I don't think the project can 
perform such a big task right now.

O.

Re: Dependency Management

[Steven Walters]

On Fri, Jun 3, 2016 at 6:20 PM, Oliver Lietz  wrote:
>
> On Monday 30 May 2016 15:28:04 Robert Munteanu wrote:
> > On Fri, 2016-05-27 at 11:14 +0200, Oliver Lietz wrote:
> > > I don't think there is support in Maven 3 for unit tests using a
> > > different
> > > version than for compile. Not sure if other build tools can handle
> > > different
> > > versions of dependencies for compile and test.
> >
> > Maven 3 is unable to do that. I can't say anything for other build
> > systems.
>
> Looks like Gradle can do (but I haven't tested):
> https://docs.gradle.org/current/userguide/java_plugin.html#sec:java_plugin_and_dependency_management

Yes, Gradle can do this.
'compile' and 'testRuntime' configurations are the standard
configuration names that are relevant here and there is a natural
inheritance chain (such that the default is that 'testRuntime'
eventually inherits everything in 'compile'),
but the versions used by 'testCompile/'testRuntime' can be overridden
to be different (usually newer) ones than what is used by
'compile'/'runtime'

I've done this plenty in the past when working with AEM to use newer
APIs/frameworks in the tests than what the code is compiled against.
such as compiling for AEM 5.6, but due to using newer versions of the
Sling mock frameworks in test cases, having newer Sling jars pulled in
as a result for test case compilation + execution.
Maven can't do this, so it makes writing tests for the same code more
difficult when using Maven when compared to Gradle under these
circumstances.

However, I don't particularly see this being enough reason alone to
warrant the effort of having Sling switching to Gradle.
There are some other possible benefits to using Gradle, such as being
able to merge separate IT modules back into modules they are testing,
therefore reducing the number of modules/build complexity.
But even so, I still don't see Sling leaving Maven.

Re: Dependency Management

[Oliver Lietz]

On Monday 30 May 2016 15:28:04 Robert Munteanu wrote:
> On Fri, 2016-05-27 at 11:14 +0200, Oliver Lietz wrote:
> > I don't think there is support in Maven 3 for unit tests using a
> > different 
> > version than for compile. Not sure if other build tools can handle
> > different 
> > versions of dependencies for compile and test.
> 
> Maven 3 is unable to do that. I can't say anything for other build
> systems.

Looks like Gradle can do (but I haven't tested):
https://docs.gradle.org/current/userguide/java_plugin.html#sec:java_plugin_and_dependency_management

> > So my preferred answers to eliminate outdated dependencies in tests:
> > 
> > 1. use minimal version for compile and current version for test
> > (how?)
> > 2. use current version for compile and test
> > 3. use current version for compile and test and carefully expand
> > import range
> 
> We seem to be discussing 'test' but I think we need to separate unit
> tests stored in the same Maven projects and integration tests.

"I don't think there is support in Maven 3 for *unit* tests using a different 
version than for compile." - it's about unit tests, sorry if it wasn't clear 
enough.

> a) Integration tests should use the same dependencies as the launchpad.
> After all, the launchpad is what we ship and things should work fine in
> that setup. If other parties want different combinations, they're free
> to define those combinations and test it themselves.

We have different setups for ITs so that is not always given (pure Pax Exam, 
based on Sling Start, external Launchpad, Pax Exam Karaf, ...).

And yes, current Launchpad should be the benchmark for new releases. Other 
parties are free to backport new releases for older systems.

> b) Unit tests may aim to use the same dependencies as the launchpad. I
> am not thoroughly convinced of the value, given that unit tests only
> touch the surface of an API. Chances of not finding out issues due to a
> missed upgrade of commons-lang are way smaller that missing an upgrade
> to oak-core ( for instance ).

AFAIR we are moving ITs closer to the code under test (Bertrand?) and we 
already have some heavy unit tests for some modules which do not have 
equivalent ITs, e.g. Event, Discovery, Distribution, Engine. Please do not 
underrate our unit tests.

Commons Collections changed behavior from 3.2.1 to 3.2.2 due to unsafe 
serialization. Similar changes can happen in other libraries for what ever 
reason. How will we notice we are affected? Reports from users?

> Also, it's quite troublesome to keep all versions of all modules in
> sync with the launchpad. All that manual work is bound to get mistakes
> for - again, IMO - little gain.
> 
> So I would say that if we want to align unit test dependencies with the
> launchpad versions we should have:
> 
> 1. An automatic way of defining dependency versions
> 2. A way of separating unit test from compile dependencies

I don't think we need to keep all modules in sync. But we should do a moderate 
modernization of (3rd party) libraries, instead of keeping backwards 
compatibility forever for the sake of itself. This leads to ugly code and we 
may miss incompatibilities with Launchpad. Again, other parties are free to 
backport new releases for older systems or upgrade libraries also (AEM 6.1 
works well with Commons Lang 3.4).

Regards,
O.

> Thanks,
> 
> Robert

Re: Dependency Management

[Robert Munteanu]

On Fri, 2016-05-27 at 09:55 +0200, Konrad Windszus wrote:
> I understand and agree with 1.
> For 2. I am not on the same page:
> 
> Why should we run ITs with the most recent version instead of the
> lowest required version?


IMO we should test with the version that is available in the launchpad
- as that's what we "ship".

Robert

> IMHO we should confirm and validate with the IT that we can really
> work with the lowest version of that dependency.
> 
> I estimate the risk for regressions in a higher version much lower,
> than the risk for facing a bug in the low version which prevents it
> from working generally.
> Konrad
> 
> 
> > On 26 May 2016, at 18:06, Julian Sedding 
> > wrote:
> > 
> > +1 to Bertrand's summary. That matches my understanding of how
> > we're
> > dealing with 3rd party dependencies in Sling.
> > 
> > 1. Keep the *required* version of a 3rd party library (an external
> > API
> > we depend upon) as low as easily practical. I.e. normally we don't
> > work around bugs or missing new APIs in 3rd party libraries, but we
> > don't raise the required version of the dependency just because it
> > is
> > available and thus artificially reducing the bundle's
> > compatibility.
> > 
> > 2. For tests or deployments (e.g. lauchpad), we aim to use the most
> > recent version of the 3rd party dependency. As that is outside of a
> > bundle's build, none of the Import-Package version ranges will be
> > affected.
> > 
> > This is a relatively simple way to create bundles with strongly
> > compatible Import-Package version ranges.
> > 
> > Regards
> > Julian
> > 
> > On Thu, May 26, 2016 at 2:46 PM, Konrad Windszus 
> > wrote:
> > > Hi,
> > > > Am 26.05.2016 um 14:36 schrieb Bertrand Delacretaz  > > > @apache.org>:
> > > > 
> > > > Hi,
> > > > 
> > > > On Thu, May 26, 2016 at 10:21 AM, Konrad Windszus  > > > .de> wrote:
> > > > > ...there was quite a discussion going on in https://issues.ap
> > > > > ache.org/jira/browse/SLING-5603 on how
> > > > > to deal with 3rd party dependencies in Sling
> > > > 
> > > > My opinion, and I think this is what we've been doing in Sling
> > > > forever
> > > > even if we didn't explicitly document it, is that if a bundle B
> > > > depends on another *for its API* then the dependency should
> > > > list the
> > > > lowest possible version.
> > > What do you mean by its API? In SLING-5603 we were talking about
> > > a dependency in the API Provider (Validation Core), not the API
> > > bundle itself.
> > > 
> > > > 
> > > > This makes B easier to use in many contexts, as long as the
> > > > right
> > > > version of the API, or a later one, is provided.
> > > > 
> > > > > ...Another aspect of option a) is: With which version of the
> > > > > dependency you want to run the IT
> > > > 
> > > > This is the "recommented implementation" aspect of the
> > > > dependency,
> > > > which is a different concern.
> > > > 
> > > > For this I suggest using the latest version of the
> > > > implementation, or
> > > > at least the one that we recommend.
> > > If we don’t test with the minimum version we depend on (and which
> > > determines the import-package version range) the risk for
> > > depending on an non-working library version is IMHO much too
> > > high. Therefore I would recommend to test with that minimum
> > > version. Testing the recommended or latest version does give less
> > > confidence that the bundle B really works with the lower version
> > > of the dependency.
> > > 
> > > 
> > > > 
> > > > -Bertrand
> > > 
> 

Re: Dependency Management

[Robert Munteanu]

On Fri, 2016-05-27 at 11:14 +0200, Oliver Lietz wrote:
> 
> I don't think there is support in Maven 3 for unit tests using a
> different 
> version than for compile. Not sure if other build tools can handle
> different 
> versions of dependencies for compile and test.

Maven 3 is unable to do that. I can't say anything for other build
systems.

> 
> So my preferred answers to eliminate outdated dependencies in tests:
> 
> 1. use minimal version for compile and current version for test
> (how?)
> 2. use current version for compile and test
> 3. use current version for compile and test and carefully expand
> import range

We seem to be discussing 'test' but I think we need to separate unit
tests stored in the same Maven projects and integration tests.

a) Integration tests should use the same dependencies as the launchpad.
After all, the launchpad is what we ship and things should work fine in
that setup. If other parties want different combinations, they're free
to define those combinations and test it themselves.

b) Unit tests may aim to use the same dependencies as the launchpad. I
am not thoroughly convinced of the value, given that unit tests only
touch the surface of an API. Chances of not finding out issues due to a
missed upgrade of commons-lang are way smaller that missing an upgrade
to oak-core ( for instance ).

Also, it's quite troublesome to keep all versions of all modules in
sync with the launchpad. All that manual work is bound to get mistakes
for - again, IMO - little gain.

So I would say that if we want to align unit test dependencies with the
launchpad versions we should have:

1. An automatic way of defining dependency versions
2. A way of separating unit test from compile dependencies

Thanks,

Robert

Re: Dependency Management

[Konrad Windszus]

> On 27 May 2016, at 11:14, Oliver Lietz  wrote:
> 
> Upgrading a library bundle to a newer version does NOT break compatibility 
> with Sling's downstream systems. It is not like upgrading from Java 7 to 8 or 
> from Declarative Services 1.2 to 1.3.
> 
> If you can deploy a newer version of a components/services bundle which runs 
> as singleton (e.g. HC Core) to an older system (e.g. AEM 6.1) you can easily 
> deploy a library bundle (whether you upgrade the older one or run both in 
> parallel). If you want to keep (for what ever reason) the older one and 
> deploying both in parallel does not work because of a limitation in OSGi 
> Installer, OSGi Installer should be improved.
I really doubt this will ever happen, because it makes things a lot more 
complicated!
My experience is different: I cannot easily upgrade the dependent 3rd party 
library bundle.

> 
> In my opinion we should always test our artifacts with current dependencies 
> (dependencies in Launchpad, a well known set). Older artifacts were already 
> tested with older dependencies.
> 
> That is currently the case for org.apache.sling.launchpad.integration-tests 
> mostly.
> 
> For unit tests and integration tests closer to the artifacts the setting is 
> different, e.g. use of Commons Lang in version 2.0, 2.2, 2.4, 2.5, 2.6, 3.0, 
> 3.0.1, 3.3.2 (similar for Commons IO, SLING-5685).
> 
> So what's the point in testing with that outdated dependencies?
> 
> I don't think there is support in Maven 3 for unit tests using a different 
> version than for compile. Not sure if other build tools can handle different 
> versions of dependencies for compile and test.
> 
> So my preferred answers to eliminate outdated dependencies in tests:
> 
> 1. use minimal version for compile and current version for test (how?)
> 2. use current version for compile and test
> 3. use current version for compile and test and carefully expand import range
> 
> Not (never) upgrading dependencies (whether our own or 3rd party) may lead to 
> nasty or unnecessary code, see Konrad's commit:
> 
> http://svn.apache.org/viewvc/sling/trunk/bundles/extensions/validation/core/src/main/java/org/apache/sling/validation/impl/ValidationServiceImpl.java?r1=1740178&r2=1745478&pathrev=1745478&diff_format=h
> 
> Why not upgrading Sling API to 2.7.0 (released two years ago) and use 
> resource.getValueMap() instead?
> 
> When do we drop support for older systems?
> 
> Regards,
> O.
> 

Re: Dependency Management

[Oliver Lietz]

Upgrading a library bundle to a newer version does NOT break compatibility 
with Sling's downstream systems. It is not like upgrading from Java 7 to 8 or 
from Declarative Services 1.2 to 1.3.

If you can deploy a newer version of a components/services bundle which runs 
as singleton (e.g. HC Core) to an older system (e.g. AEM 6.1) you can easily 
deploy a library bundle (whether you upgrade the older one or run both in 
parallel). If you want to keep (for what ever reason) the older one and 
deploying both in parallel does not work because of a limitation in OSGi 
Installer, OSGi Installer should be improved.

In my opinion we should always test our artifacts with current dependencies 
(dependencies in Launchpad, a well known set). Older artifacts were already 
tested with older dependencies.

That is currently the case for org.apache.sling.launchpad.integration-tests 
mostly.

For unit tests and integration tests closer to the artifacts the setting is 
different, e.g. use of Commons Lang in version 2.0, 2.2, 2.4, 2.5, 2.6, 3.0, 
3.0.1, 3.3.2 (similar for Commons IO, SLING-5685).

So what's the point in testing with that outdated dependencies?

I don't think there is support in Maven 3 for unit tests using a different 
version than for compile. Not sure if other build tools can handle different 
versions of dependencies for compile and test.

So my preferred answers to eliminate outdated dependencies in tests:

1. use minimal version for compile and current version for test (how?)
2. use current version for compile and test
3. use current version for compile and test and carefully expand import range

Not (never) upgrading dependencies (whether our own or 3rd party) may lead to 
nasty or unnecessary code, see Konrad's commit:

http://svn.apache.org/viewvc/sling/trunk/bundles/extensions/validation/core/src/main/java/org/apache/sling/validation/impl/ValidationServiceImpl.java?r1=1740178&r2=1745478&pathrev=1745478&diff_format=h

Why not upgrading Sling API to 2.7.0 (released two years ago) and use 
resource.getValueMap() instead?

When do we drop support for older systems?

Regards,
O.

Re: Dependency Management

[Carsten Ziegeler]

Konrad Windszus wrote
> I am strongly in favor of letting bnd generate the Import-Package headers 
> (incl. version ranges) and to not tweak those manually.
> Manually adjusting the process is IMHO very error-prone.
+5

Carsten


 
-- 
Carsten Ziegeler
Adobe Research Switzerland
cziege...@apache.org

Re: Dependency Management

[Konrad Windszus]

I am strongly in favor of letting bnd generate the Import-Package headers 
(incl. version ranges) and to not tweak those manually.
Manually adjusting the process is IMHO very error-prone.
Konrad

> On 26 May 2016, at 18:13, Julian Sedding  wrote:
> 
> Addendum: I just read Olli's comment in the ticket.
> 
> My requirement is that we keep the Import_package version ranges of
> bundles flexible in order not to require artificially high 3rd party
> dependency versions. How we achieve this is up for debate and
> ultimately a tooling issue (by default bnd generates the
> Import-Package version ranges based on the dependencies on the build
> class-path).
> 
> Regards
> Julian
> 
> 
> On Thu, May 26, 2016 at 6:06 PM, Julian Sedding  wrote:
>> +1 to Bertrand's summary. That matches my understanding of how we're
>> dealing with 3rd party dependencies in Sling.
>> 
>> 1. Keep the *required* version of a 3rd party library (an external API
>> we depend upon) as low as easily practical. I.e. normally we don't
>> work around bugs or missing new APIs in 3rd party libraries, but we
>> don't raise the required version of the dependency just because it is
>> available and thus artificially reducing the bundle's compatibility.
>> 
>> 2. For tests or deployments (e.g. lauchpad), we aim to use the most
>> recent version of the 3rd party dependency. As that is outside of a
>> bundle's build, none of the Import-Package version ranges will be
>> affected.
>> 
>> This is a relatively simple way to create bundles with strongly
>> compatible Import-Package version ranges.
>> 
>> Regards
>> Julian
>> 
>> On Thu, May 26, 2016 at 2:46 PM, Konrad Windszus  wrote:
>>> Hi,
 Am 26.05.2016 um 14:36 schrieb Bertrand Delacretaz 
 :
 
 Hi,
 
 On Thu, May 26, 2016 at 10:21 AM, Konrad Windszus  wrote:
> ...there was quite a discussion going on in 
> https://issues.apache.org/jira/browse/SLING-5603 on how
> to deal with 3rd party dependencies in Sling
 
 My opinion, and I think this is what we've been doing in Sling forever
 even if we didn't explicitly document it, is that if a bundle B
 depends on another *for its API* then the dependency should list the
 lowest possible version.
>>> What do you mean by its API? In SLING-5603 we were talking about a 
>>> dependency in the API Provider (Validation Core), not the API bundle itself.
>>> 
 
 This makes B easier to use in many contexts, as long as the right
 version of the API, or a later one, is provided.
 
> ...Another aspect of option a) is: With which version of the dependency 
> you want to run the IT
 
 This is the "recommented implementation" aspect of the dependency,
 which is a different concern.
 
 For this I suggest using the latest version of the implementation, or
 at least the one that we recommend.
>>> If we don’t test with the minimum version we depend on (and which 
>>> determines the import-package version range) the risk for depending on an 
>>> non-working library version is IMHO much too high. Therefore I would 
>>> recommend to test with that minimum version. Testing the recommended or 
>>> latest version does give less confidence that the bundle B really works 
>>> with the lower version of the dependency.
>>> 
>>> 
 
 -Bertrand
>>> 

Re: Dependency Management

[Konrad Windszus]

I understand and agree with 1.
For 2. I am not on the same page:

Why should we run ITs with the most recent version instead of the lowest 
required version?
IMHO we should confirm and validate with the IT that we can really work with 
the lowest version of that dependency.

I estimate the risk for regressions in a higher version much lower, than the 
risk for facing a bug in the low version which prevents it from working 
generally.
Konrad


> On 26 May 2016, at 18:06, Julian Sedding  wrote:
> 
> +1 to Bertrand's summary. That matches my understanding of how we're
> dealing with 3rd party dependencies in Sling.
> 
> 1. Keep the *required* version of a 3rd party library (an external API
> we depend upon) as low as easily practical. I.e. normally we don't
> work around bugs or missing new APIs in 3rd party libraries, but we
> don't raise the required version of the dependency just because it is
> available and thus artificially reducing the bundle's compatibility.
> 
> 2. For tests or deployments (e.g. lauchpad), we aim to use the most
> recent version of the 3rd party dependency. As that is outside of a
> bundle's build, none of the Import-Package version ranges will be
> affected.
> 
> This is a relatively simple way to create bundles with strongly
> compatible Import-Package version ranges.
> 
> Regards
> Julian
> 
> On Thu, May 26, 2016 at 2:46 PM, Konrad Windszus  wrote:
>> Hi,
>>> Am 26.05.2016 um 14:36 schrieb Bertrand Delacretaz :
>>> 
>>> Hi,
>>> 
>>> On Thu, May 26, 2016 at 10:21 AM, Konrad Windszus  wrote:
 ...there was quite a discussion going on in 
 https://issues.apache.org/jira/browse/SLING-5603 on how
 to deal with 3rd party dependencies in Sling
>>> 
>>> My opinion, and I think this is what we've been doing in Sling forever
>>> even if we didn't explicitly document it, is that if a bundle B
>>> depends on another *for its API* then the dependency should list the
>>> lowest possible version.
>> What do you mean by its API? In SLING-5603 we were talking about a 
>> dependency in the API Provider (Validation Core), not the API bundle itself.
>> 
>>> 
>>> This makes B easier to use in many contexts, as long as the right
>>> version of the API, or a later one, is provided.
>>> 
 ...Another aspect of option a) is: With which version of the dependency 
 you want to run the IT
>>> 
>>> This is the "recommented implementation" aspect of the dependency,
>>> which is a different concern.
>>> 
>>> For this I suggest using the latest version of the implementation, or
>>> at least the one that we recommend.
>> If we don’t test with the minimum version we depend on (and which determines 
>> the import-package version range) the risk for depending on an non-working 
>> library version is IMHO much too high. Therefore I would recommend to test 
>> with that minimum version. Testing the recommended or latest version does 
>> give less confidence that the bundle B really works with the lower version 
>> of the dependency.
>> 
>> 
>>> 
>>> -Bertrand
>> 

Re: Dependency Management

[Julian Sedding]

Addendum: I just read Olli's comment in the ticket.

My requirement is that we keep the Import_package version ranges of
bundles flexible in order not to require artificially high 3rd party
dependency versions. How we achieve this is up for debate and
ultimately a tooling issue (by default bnd generates the
Import-Package version ranges based on the dependencies on the build
class-path).

Regards
Julian


On Thu, May 26, 2016 at 6:06 PM, Julian Sedding  wrote:
> +1 to Bertrand's summary. That matches my understanding of how we're
> dealing with 3rd party dependencies in Sling.
>
> 1. Keep the *required* version of a 3rd party library (an external API
> we depend upon) as low as easily practical. I.e. normally we don't
> work around bugs or missing new APIs in 3rd party libraries, but we
> don't raise the required version of the dependency just because it is
> available and thus artificially reducing the bundle's compatibility.
>
> 2. For tests or deployments (e.g. lauchpad), we aim to use the most
> recent version of the 3rd party dependency. As that is outside of a
> bundle's build, none of the Import-Package version ranges will be
> affected.
>
> This is a relatively simple way to create bundles with strongly
> compatible Import-Package version ranges.
>
> Regards
> Julian
>
> On Thu, May 26, 2016 at 2:46 PM, Konrad Windszus  wrote:
>> Hi,
>>> Am 26.05.2016 um 14:36 schrieb Bertrand Delacretaz :
>>>
>>> Hi,
>>>
>>> On Thu, May 26, 2016 at 10:21 AM, Konrad Windszus  wrote:
 ...there was quite a discussion going on in 
 https://issues.apache.org/jira/browse/SLING-5603 on how
 to deal with 3rd party dependencies in Sling
>>>
>>> My opinion, and I think this is what we've been doing in Sling forever
>>> even if we didn't explicitly document it, is that if a bundle B
>>> depends on another *for its API* then the dependency should list the
>>> lowest possible version.
>> What do you mean by its API? In SLING-5603 we were talking about a 
>> dependency in the API Provider (Validation Core), not the API bundle itself.
>>
>>>
>>> This makes B easier to use in many contexts, as long as the right
>>> version of the API, or a later one, is provided.
>>>
 ...Another aspect of option a) is: With which version of the dependency 
 you want to run the IT
>>>
>>> This is the "recommented implementation" aspect of the dependency,
>>> which is a different concern.
>>>
>>> For this I suggest using the latest version of the implementation, or
>>> at least the one that we recommend.
>> If we don’t test with the minimum version we depend on (and which determines 
>> the import-package version range) the risk for depending on an non-working 
>> library version is IMHO much too high. Therefore I would recommend to test 
>> with that minimum version. Testing the recommended or latest version does 
>> give less confidence that the bundle B really works with the lower version 
>> of the dependency.
>>
>>
>>>
>>> -Bertrand
>>

Re: Dependency Management

[Julian Sedding]

+1 to Bertrand's summary. That matches my understanding of how we're
dealing with 3rd party dependencies in Sling.

1. Keep the *required* version of a 3rd party library (an external API
we depend upon) as low as easily practical. I.e. normally we don't
work around bugs or missing new APIs in 3rd party libraries, but we
don't raise the required version of the dependency just because it is
available and thus artificially reducing the bundle's compatibility.

2. For tests or deployments (e.g. lauchpad), we aim to use the most
recent version of the 3rd party dependency. As that is outside of a
bundle's build, none of the Import-Package version ranges will be
affected.

This is a relatively simple way to create bundles with strongly
compatible Import-Package version ranges.

Regards
Julian

On Thu, May 26, 2016 at 2:46 PM, Konrad Windszus  wrote:
> Hi,
>> Am 26.05.2016 um 14:36 schrieb Bertrand Delacretaz :
>>
>> Hi,
>>
>> On Thu, May 26, 2016 at 10:21 AM, Konrad Windszus  wrote:
>>> ...there was quite a discussion going on in 
>>> https://issues.apache.org/jira/browse/SLING-5603 on how
>>> to deal with 3rd party dependencies in Sling
>>
>> My opinion, and I think this is what we've been doing in Sling forever
>> even if we didn't explicitly document it, is that if a bundle B
>> depends on another *for its API* then the dependency should list the
>> lowest possible version.
> What do you mean by its API? In SLING-5603 we were talking about a dependency 
> in the API Provider (Validation Core), not the API bundle itself.
>
>>
>> This makes B easier to use in many contexts, as long as the right
>> version of the API, or a later one, is provided.
>>
>>> ...Another aspect of option a) is: With which version of the dependency you 
>>> want to run the IT
>>
>> This is the "recommented implementation" aspect of the dependency,
>> which is a different concern.
>>
>> For this I suggest using the latest version of the implementation, or
>> at least the one that we recommend.
> If we don’t test with the minimum version we depend on (and which determines 
> the import-package version range) the risk for depending on an non-working 
> library version is IMHO much too high. Therefore I would recommend to test 
> with that minimum version. Testing the recommended or latest version does 
> give less confidence that the bundle B really works with the lower version of 
> the dependency.
>
>
>>
>> -Bertrand
>

Re: Dependency Management

[Konrad Windszus]

Hi,
> Am 26.05.2016 um 14:36 schrieb Bertrand Delacretaz :
> 
> Hi,
> 
> On Thu, May 26, 2016 at 10:21 AM, Konrad Windszus  wrote:
>> ...there was quite a discussion going on in 
>> https://issues.apache.org/jira/browse/SLING-5603 on how
>> to deal with 3rd party dependencies in Sling
> 
> My opinion, and I think this is what we've been doing in Sling forever
> even if we didn't explicitly document it, is that if a bundle B
> depends on another *for its API* then the dependency should list the
> lowest possible version.
What do you mean by its API? In SLING-5603 we were talking about a dependency 
in the API Provider (Validation Core), not the API bundle itself.

> 
> This makes B easier to use in many contexts, as long as the right
> version of the API, or a later one, is provided.
> 
>> ...Another aspect of option a) is: With which version of the dependency you 
>> want to run the IT
> 
> This is the "recommented implementation" aspect of the dependency,
> which is a different concern.
> 
> For this I suggest using the latest version of the implementation, or
> at least the one that we recommend.
If we don’t test with the minimum version we depend on (and which determines 
the import-package version range) the risk for depending on an non-working 
library version is IMHO much too high. Therefore I would recommend to test with 
that minimum version. Testing the recommended or latest version does give less 
confidence that the bundle B really works with the lower version of the 
dependency.


> 
> -Bertrand

Re: Dependency Management

[Bertrand Delacretaz]

Hi,

On Thu, May 26, 2016 at 10:21 AM, Konrad Windszus  wrote:
> ...there was quite a discussion going on in 
> https://issues.apache.org/jira/browse/SLING-5603 on how
> to deal with 3rd party dependencies in Sling

My opinion, and I think this is what we've been doing in Sling forever
even if we didn't explicitly document it, is that if a bundle B
depends on another *for its API* then the dependency should list the
lowest possible version.

This makes B easier to use in many contexts, as long as the right
version of the API, or a later one, is provided.

> ...Another aspect of option a) is: With which version of the dependency you 
> want to run the IT

This is the "recommented implementation" aspect of the dependency,
which is a different concern.

For this I suggest using the latest version of the implementation, or
at least the one that we recommend.

-Bertrand