As long as the API defines that the adapt() method may return null, adding an
adapt interface does not violate the provider's backward compability. Since it
does not know the interface, it must always return null. So adding an interface
is not required to be implemented by the provider it could be the first
reasonable API micro case ... However ...
I think your reasoning is wrong about the consumer getting the newbie, it won't
always.
C [1.0.1,2) -> A 1.0.1 <- [1,1.1) P
Since P is allowed to be oblivious of changes in A between 1.0.0 and 1.0.1
(micro update) it cannot be required to provide the new feature, hence the
runtime check in the consumer. So C's requirement for > 1.0.1 is only to ensure
it can see the new interface but does not say anything about P.
Another aspect. The default bnd version policy for consumers ignores the micro
for the base. I've done this because changes then version constraints become
very rigid otherwise and you get largely large and unnecessary upgrades of all
transiently affected bundles. It is a bit of 'oil'. So by default bnd would do
C.A[1,2) even if C was compiled against 1.0.1. This would allow it to bind to A
1.0.0 and thus create a class not found exception. It goes without saying
(hopefully) that you can set your own version policy for this in bnd.
Though I initially thought this was the first reasonable use case for a micro
upgrade of an API, I think it does seems to be better to make it a minor update
although this kills the provider's compatibility.
Kind regards,
Peter Kriens
On 2 feb. 2013, at 05:03, James Watkins-Harvey wrote:
> Henning,
>
>
> I'm glad that my suggestions fit your issues.
>
>
> Regarding your questions, let me begin with the second one. You should really
> never fork your API. As Neil said it earlier, APIs should only ever move
> forwards. Never make compromise on that one, or you will be back to your
> original issue. So only implementation bundles may be forked, and then again,
> only if the specific situation justify to do so. That is always the very last
> card in my hand, to be handled only if every other solutions has been
> discarded. Forked bundles means duplicated code, which means maintenance
> costs. If you let a fork go on for too long, the code diverge to a point
> where that maintenance cost clearly outweigh the advantages you may have
> gained originally. But in some development context, the business reality
> requires these extreme solutions.
>
>
> Lets move on to the first question. The "add-on interfaces" may technically
> be created anywhere (that is, inside any exported package, be it in the same
> bundle or a different bundle). In fact, putting them in a different bundle
> could be justified, if the "add-on" represents a complement API, rather than
> a bare upgrade to the original API; note however that I have never faced such
> a case in real life, and that I struggle right now in proposing an easily
> understandable example of such case.
>
> More realistically, the common use case is to extend an existing API. That
> is, the "add-on interface" will then be a part of the API. For that reason,
> it makes sense that the add-on interface be located inside the same package
> than the original interface, and given what has been said previously, in the
> same non-forked API bundle. For my team, it is a convention to create them as
> nested interfaces, inside the original interface, as this makes it easier for
> developers of implementations to locate the set of potential upgrades. Note
> that creating nested members does not break binary compatibility for neither
> producers nor consumers of the containing member.
>
> The key point in extending the API though "add-on interfaces" is that it
> allows for extending the API without breaking neither producers nor
> consumers: compliance with the newer version of the API is optional, even for
> producers. This is a design compromise, though, made at the expense of
> consumers, where you may have to deal with a few "what-if-feature-x-is-not
> -supported". That is certainly not a one-size-fit-all approach, but I know of
> several projects that do so, especially for interfaces that are meant to be
> consumer-implemented. Have a look for example at all classes named
> "I.*Extension[0-9]*$" in Eclipse API (there are quite a few in
> "org.eclipse.jface.text**").
>
> What is the impact of add-on interfaces on packages/bundles versioning?
> Adding a new interface inside a package is definitely an upgrade to that
> package, and therefore requires a modification to the API's package (and
> bundle) version number. Given that the change does not break compatibility
> with neither existing producers nor existing consumers, it could rightfully
> (given a strict reading of the Semantic Versioning whitepaper) be said that
> only the micro part needs to be increased. For example, if I initially had
> interface IProductStore in c.x.productstores v1.0.0, and then added interface
> IManageableProductStore, the c.x.productstores could then be versionized as
> 1.0.1; the bundle would similarly be upgraded to the next micro. Producers
> that intend to support the extended feature would then have to require
> c.x.productstores [1.0.1,1.1). Consumers that intend to use the extended
> feature would require c.x.productstores [1.0.1,2.0).
>
> Alternatively, it might be argued that any change to the exposed API (even
> changes that don't breaks compatibility with existing users) justify a minor
> upgrade. This restriction ensure that a bundle may always be substituted by a
> previous release with the same major and minor. That is the convention my
> team has been using till now. Given that convention, after adding
> IManageableProductStore in the previous example, the package
> c.x.productstores would be versionized as 1.1.0; the bundle would similarly
> be upgraded to the next minor. Producers that intend to support the extended
> feature would then have to require c.x.productstores [1.1,2.0), assuming that
> the package is guaranteed to only ever be upgraded in a way that do not break
> compatibility with existing producers. Consumers that intend to use the
> extended feature would, similarly, require c.x.productstores [1.1,2.0).
>
> Now that I have been using this strategy for some time, I have to say that
> the second strategy is somewhat more restrictive, and there is very little
> real advantage in the backward substitutability of bundles within the same
> major.minor. For this reason, should you decide to go on with this
> proposition, I would recommend that you go with the first versioning policy.
>
>
> Hope that clarifies everything,
>
>
> James Watkins-Harvey
>
>
>
> On 2013-01-30, at 16h32, Henning Andersen wrote:
>
>> Hi all,
>>
>> thanks for all your comments, suggestions and experience, it is highly
>> appreciated! A few follow-up questions:
>>
>> @James:
>>
>> The issues you are facing sound very familiar to ours. I like the fact that
>> you have several options and I think you are right in that there is no
>> single solution to the issues. Thanks for the elaborate response and I hope
>> you do not mind a few questions:
>>
>> Re item 2, "set-of-features" add-on interfaces: do you also do this in
>> branches? If yes, am I correct in that you then must put the new interface
>> in a new package to be able to give it a proper version?
>>
>> Re item 3, forking bundles: when you fork a bundle for a specific customer,
>> into: c.x.somebundle.customizations.custumerabc, do you only do so for pure
>> implementation bundles or also for bundles with APIs? If also for APIs, how
>> do you handle the version numbers of package exports?
>>
>>
>> @Neil:
>>
>> Thanks for the tip on bndtools/bnd. I see the new commandline options
>> (baseline and diff plus some I did not try out) in bnd and it does have a
>> bit of help built-in. I wonder if there is more documentation available
>> anywhere (seems the aqute.biz site does not contain any info on this and
>> neither does the "bnd-book"). Specifically I miss info on:
>>
>> 1. How to indicate that an interface is consumer implemented (ie. any
>> extension is also a major change)?
>> 2. Whether it is possible to add an indication to the package that a new
>> minor or major version is necessary due to changing semantics of
>> implementation/javadoc specification (so a CI server can later see it).
>> 3. Any plugin capabilities available to override some of the behaviour?
>>
>> I could of course read the code to find out!
>>
>>
>> Best regards,
>>
>> Henning
>>
>>
>>
>> On Tue, Jan 29, 2013 at 11:11 PM, James Watkins-Harvey
>> <[email protected]> wrote:
>> Hi Henning,
>>
>>
>> My team also face such fast-pace evolution that sometime impose quick
>> modifications to APIs. The real problem is not the lack of thinking we put
>> in designing these extended API, but the fact that these API extensions
>> might be required at a time when it would be unacceptable to force immediate
>> upgrade of all related bundles. Of course, simply not allowing unordered
>> upgrades is, from a technical point of view, the best and simplest approach.
>> But sometimes, the pressure to quickly provide a "low risk" upgrade to
>> existing feature is simply too high, be it the "time to market" factor, or
>> whatever situation we may have with a specific customer. In our case, being
>> able to do so is actually a business requirement.
>>
>> Let me share with you an overview of how some development practices we have
>> established in order to better to deal with these situations. The key
>> strategies are:
>> Heavily split bundles in API vs Implementations bundles;
>> Design "set-of-features" add-on interfaces whenever a fundamental interface
>> need to be upgraded;
>> Fork projects when a bundle is subject to external evolution factors (or has
>> to be versionized simultaneously on two axis);
>> Testing API is even more important than testing implementations.
>> The first point states that, in most case, a bundle should either offer an
>> API or an implementation, but not both at the same time. I say in most
>> cases, though, because "utilities" bundles and some clearly delimited
>> bundles might not need such precautions. But not splitting enough tends to
>> hurt more than splitting too much, so in case of doubts, better split.
>>
>> API bundles (and packages…) are semantically versionnized. API classes and
>> interfaces grow from a minor to the next one, and can eventually be
>> consolidated at major, for example to fix up any mistakes that might have
>> been introduced over time, or to remove methods that have become deprecated.
>> Note that given the semantic versioning rules, API bundles can almost never
>> be altered at micro level.
>>
>> The main down side of isolating API is that it becomes impossible for a
>> bundle to directly instantiate (or extends) classes provided by
>> implementation bundles. This is easily fixed however by defining
>> factory/repository/manager/whatever else interfaces in the API, and having
>> the implementation bundle register its implementation of that interface as a
>> service. As for inheritance, it is considered a bad practice anyway.
>>
>>
>> Let's move on to the second point: we must be very strict in how API
>> interfaces and classes evolve. Basically, we consider that most API
>> interfaces are unalterable within the scope of a minor upgrade. This is a
>> common best practice whenever an interface is intended to be implemented by
>> several external bundles, because it may be impossible to track with
>> certainty all implementations of the interface, which therefore present risk
>> of binary incompatibilities at runtime.
>>
>> However, we also consider as unalterable many interfaces not intended to be
>> publicly implemented, such as interfaces of services exposed through OSGi.
>> Instead, when new methods are required, we define a new interface that
>> extends the original interface (in general, we define those within the scope
>> of the original interface, simply to make it easier to track all extensions
>> that have been developed). For example, if we defined interface
>> "IProductStore" in the API bundle c.x.productstores v1.0.0, then we shall
>> not alter that interface before v2.0.0 of the API. If we eventually need to
>> add more methods in order to support a new feature, then we would define
>> them in a new interface, say IManageableProductStore (that extends
>> IProductStore). That would be API bundle c.x.productstores v.1.1.0. Then in
>> v1.2.0, we might have ISearcheableProductStore (also extending
>> IProductStore), which would introduce two more methods.
>>
>> The real gain in doing so is that we might independently delay upgrading to
>> a specific feature, on both side of the API (that is, implementers and
>> users). For example, if for some reason, we consider that one implementation
>> of IProductStore can't easily be transitioned to support the
>> IManageableProductStore, then it may still support the
>> ISearcheableProductStore interface. Obviously, this push some of the
>> difficulties of upgrades back to the code using the API, since programmers
>> must now be careful to check if a given object implements the extended
>> interface before invoking some extended feature. However, this is generally
>> a very descent trade off: first, there are in general few users of those
>> extended API (otherwise, it is probable that the methods would have been
>> identified early during development). Second, this is new code anyway, so
>> all places where checks has to be performed are known anyway. And third,
>> there are tons of very nice idioms to efficiently deal with these "test
>> before acting" sequences.
>>
>> Note that this solution can trivially be implemented using the "instance of"
>> operator, followed by casting objects to the extended class. However, we
>> prefer to define a method "adapt(Class<X> clazz) : X" in our base
>> interfaces, which allow more dynamic decisions on either a given feature
>> should be supported or not. In the previous example, a specific
>> implementation of IProductStore might accept to be adapted to
>> IManageableProductStore only if the connected user possess administration
>> rights. Or the code might look at company-level settings to decide either a
>> given feature is enabled or not. This is both cleaner and safer than having
>> the code using the API deal with permissions and "company-wide" settings.
>>
>>
>> Now up to the third point… As we described earlier, mapping evolution of an
>> API bundle in a semantic versioning scheme is straight forward. In general,
>> implementation bundles can also be trivially mapped into semantic versioning
>> scheme (I won't elaborate here). However, we note that in some situations,
>> the versioning of a bundle may be affected not only by internal corrections
>> and API additions, but also by external factors.
>>
>> Such external factors are common when a bundle ensure compatibility with an
>> external subsystem: there might be several major version of a key external
>> application, the customer database may be frozen to an older schema version,
>> and so on. In simple case, these external factors might easily be dealt by
>> "if" statements and dynamic checks at runtime, or possibly by having a few
>> distinct classes inside a same bundle. However, in more complex situations,
>> it may be tempting to produce distinct versions of the bundle. To illustrate
>> this case, let assume that bundle c.x.fileimport.appa provides a connector
>> to read files produced by an external program named "Application A - 1.0".
>> Now if that application file format changes in release 2.0, one might think
>> that the bundle should equivalently be upgraded to "2.0". Yet, assuming that
>> support for file format 1 and file format 2 can't reasonably be implemented
>> in the same bundle, then changing the bundle version number would be
>> inappropriate: doing so would prevent the bundle for format 1 from later
>> being upgraded to a new major API, since that would require that the
>> implementation bundle's version be also upgraded to the next major number,
>> which is no longer available. In these cases, the correct approach (still
>> assuming that both formats can't reasonably be implemented in a same bundle)
>> would be to fork the original bundle into a second one, with a distinct id
>> and a distinct version number. That could be for example
>> c.x.fileimport.appa.format2 or c.x.fileimport.appa_v2.
>>
>> Hopefully, this example is easily understood and accepted. However, we have
>> found that the same logic holds in many cases that are much less trivial.
>> For example, for reasons that I won't elaborate here, our database schemas
>> evolution is not coordinated with our main software releases. This means
>> that at the time the software connects to the database, the schema version
>> is checked, and then the bundle specific to the schema version is used
>> thereafter. However, because evolution of the implementation bundles happens
>> on two axes (specifically, the API version and the schema version), it would
>> be problematic to express semantic evolution through the classical version
>> numbers. Therefore, we create distinct bundles for each schema version,
>> named something like c.x.persistence.ourdatabase._3_141 4.2.5, where the
>> schema version is 3.141, the implemented API is 4.2, and there has been 5
>> corrections to the code since the last upgrade to the API.
>>
>> This strategy can be extended to other cases where external factors make it
>> impossible to allow a bundle evolution by following a simpler path. For
>> example, we sometimes judge that, for exceptional reasons, it is preferable
>> for a specific bundle to be temporarily forked for the needs of a specific
>> customer. Doing this produce a totally distinct bundle, let's say
>> c.x.somebundle.customizations.custumerabc, with distinct version numbers.
>> This is possible because anyway, no one has direct requirements for the
>> original bundle (c.x.somebundle.impl).
>>
>> Obviously, systematic forking has some drawback, most notably that it makes
>> the code harder to maintain; fortunately, these forks are generally very
>> short lived. For example, we rarely have to support more than 2 or 3 schema
>> versions backward, and we only allow a customer specific branch to live
>> until the end of a "customer production cycle" (which rarely span more than
>> 3 months, during which we have to minimize risks for that specific
>> customer). As soon as that customer's production cycle is completed, we
>> transition him back to the main branch.
>>
>>
>> Together, the previous strategies ensure that we always have a few possible
>> solutions to deal with customer specific evolutions. We may for example fork
>> bundles (usually not the preferred path, but it is sometimes unavoidable),
>> or introduce new configuration items (which may be a good approach when we
>> believe that several customers might be interested in that same
>> customization, and it appears that the customization will remain pertinent
>> in the long term), or define a cleaner customization end point (if the
>> customization appears legitimate in the long term, and we believe that
>> similar customizations will be required for other customers), and so on.
>> Also, even if we choose to introduce a new, customer specific bundle, we
>> might prefer to do so as a fragment, or to avoid code duplication by having
>> the replacement service (registered with an higher ranking) capturing the
>> service exported by the original implementation bundle, and then delegate or
>> decorate method calls to the original service (rather than reimplement all
>> the code).
>>
>> In all case, we must carefully weight the pros and cons of each solution for
>> any specific issue. Maintainability and correctness are both majors issues
>> with these customizations. For this reason, I always question the time to
>> live of a customization and the probability that both the original code and
>> the customized code will have to evolve again during that time span.
>>
>>
>> The last point basically states that we should always have
>> implementation-independent tests, that validate that implementations do
>> indeed honour the API contract. It is obviously not always possible to test
>> all aspect of an implementation bundle in this way, but these
>> implementation-independant tests are particularly important once we deal
>> with multiple forks of a base implementation. Indeed, back porting tests
>> between each customer-specific branches of a base implementation is
>> cumbersome. Testing to API definitely reduce the maintenance cost of
>> customer specific branches.
>>
>>
>>
>> I hope this helps,
>>
>> James Watkins-Harvey
>>
>>
>>
>>
>> On 2013-01-25, at 09h54, Henning Andersen wrote:
>>
>>> Hi OSGi developers,
>>>
>>> we have been using OSGi for 3 years now, however without semantic
>>> versioning. We now want to take our versioning strategy to the next level
>>> by introducing semantic versioning and allowing individual release cycles
>>> for different components in our system.
>>>
>>> One of the issues we are struggling with is how to handle branches and
>>> versioning, especially in relation to API changes.
>>>
>>> Consider a package p with a single class C, with a single method m:
>>>
>>> package p;
>>>
>>> public class C {
>>> public void m();
>>> }
>>>
>>> Now say, we released a bundle B in version 1.0, with this package in
>>> version 1.0. To allow us to bug fix, the bundle B is branched into a 1.0
>>> branch too in our source code repository.
>>>
>>> Another bundle X has Import-Package: p; version="[1.0;2)" as per semantic
>>> versioning and is released as version 1.0.
>>>
>>> Development of class C continues and a new method is added:
>>>
>>> public class C {
>>> public void m();
>>> public void n();
>>> }
>>>
>>> This is released in a new version of B, bundle-version 1.1, package version
>>> 1.1.
>>>
>>> Now a customer has X 1.0 and B 1.0. They request a new feature to X, but
>>> reject to upgrade B. The feature in X requires an API extension to the
>>> class C. So we add this in the 1.0 branch:
>>>
>>> public class C {
>>> public void m();
>>> public void o();
>>> }
>>>
>>> If we were to follow semantic versioning, we should release this as B
>>> version 1.1, p version 1.1. However, these numbers are already occupied.
>>>
>>> So we have to break the principle and release them as B version 1.0.1 and p
>>> version 1.0.1.
>>>
>>> The developer of X now needs to import the package p in the right version.
>>> However, using: Import-Package: p; version="[1.0.1;2)" is not entirely
>>> right, since his code is not compatible with the 1.1 version.
>>>
>>> So far we have 5 solutions:
>>>
>>> 1. Do not allow it. Only allow extending the API in a new minor version
>>> following the latest released version.
>>> 2. Add an attribute to the export 'Export-Package: a; version=1.0.1; p.C.o
>>> = true' and do the same on 'Import-Package: a; version="[1.0.1,2)"; p.C.o =
>>> true'. Requires that we identify the dependency to method level.
>>> 3. Fix the version of a in X: 'Import-Package: a; version="[1.0.1,1.1)"'.
>>> This however has the implication that X need upgrade when we upgrade B.
>>> 4. Build two bundles out of X, one with 'Import-Package: a;
>>> version="[1.0.1,1.1)"' and one with 'Import-Package: a;
>>> version="[1.1.1,2)"' (assuming the method was also added in 1.1.1).
>>> 5. Branch X into two versions (much like 3), with different import versions
>>>
>>> We think some of these could work, but do wonder if others have run into
>>> the same issues or similar branching/versioning issues and how they solved
>>> it? Any input is highly appreciated.
>>>
>>> Thanks,
>>>
>>> Henning
>>>
>>>
>>> _______________________________________________
>>> OSGi Developer Mail List
>>> [email protected]
>>> https://mail.osgi.org/mailman/listinfo/osgi-dev
>>
>>
>> _______________________________________________
>> OSGi Developer Mail List
>> [email protected]
>> https://mail.osgi.org/mailman/listinfo/osgi-dev
>>
>> _______________________________________________
>> OSGi Developer Mail List
>> [email protected]
>> https://mail.osgi.org/mailman/listinfo/osgi-dev
>
> _______________________________________________
> OSGi Developer Mail List
> [email protected]
> https://mail.osgi.org/mailman/listinfo/osgi-dev
_______________________________________________
OSGi Developer Mail List
[email protected]
https://mail.osgi.org/mailman/listinfo/osgi-dev
