I haven't closely followed the discussion so far, but isn't it Apache policy that major versions should stay backwards compatible to all previous releases with the same major version?
-s 2017-06-28 12:26 GMT+02:00 Kostas Kloudas <k.klou...@data-artisans.com>: > I agree that 1.1 compatibility is the most important “pain point", as > compatibility with the rest of the versions follows a more “systematic” > approach. > > I think that discarding compatibility with 1.1 will clear some parts > of the codebase significantly. > > Kostas > > > On Jun 27, 2017, at 6:03 PM, Stephan Ewen <se...@apache.org> wrote: > > > > I think that this discussion is probably motivated especially by the > > "legacy state" handling of Flink 1.1. > > The biggest gain in codebase and productivity would be won only by > dropping > > 1.1 compatibility in Flink 1.4. > > > > My gut feeling is that this is reasonable. We support two versions back, > > which means that users can skip one upgrade, but not two. > > > > From what I can tell, users are usually eager to upgrade. They don't do > it > > immediately, but as soon as the new release is a bit battle tested. > > > > I would expect skipping two entire versions to be rare enough to be okay > > with a solution which is a bit more effort for the user: > > You can upgrade from Flink 1.1. to 1.4 by loading the 1.1 savepoint into > > Flink 1.2, take a savepoint (1.2 format), and resume that in Flink 1.4. > > > > Greetings, > > Stephan > > > > > > On Tue, Jun 27, 2017 at 12:01 PM, Stefan Richter < > > s.rich...@data-artisans.com> wrote: > > > >> For many parts of the code, I would agree with Aljoscha. However, I can > >> also see notable exceptions, such as maintaining support for the legacy > >> state from Flink <=1.1. For example, I think dropping support for this > can > >> simplify new developments such as fast local recovery or state > replication > >> quiet a bit because this is a special case that runs through a lot of > code > >> from backend to JM. So besides this general discussion about a backwards > >> compatible policy, do you think it could make sense to start another > >> concrete discussion about if we still must or want backwards > compatibility > >> to Flink 1.1 in Flink 1.4? > >> > >>> Am 29.05.2017 um 12:08 schrieb Aljoscha Krettek <aljos...@apache.org>: > >>> > >>> Normally, I’m the first one to suggest removing everything that is not > >> absolutely necessary in order to have a clean code base. On this issue, > >> though, I think we should support restoring from old Savepoints as far > back > >> as possible if it does not make the code completely unmaintainable. Some > >> users might jump versions and always forcing them to go though every > >> version from their old version to the current version doesn’t seem > feasible > >> and might put off some users. > >>> > >>> So far, I think the burden of supporting restore from 1.1 is still > small > >> enough and with each new version the changes between versions become > less > >> and less. The changes from 1.2 to the upcoming 1.3 are quite minimal, I > >> think. > >>> > >>> Best, > >>> Aljoscha > >>>> On 24. May 2017, at 17:58, Ted Yu <yuzhih...@gmail.com> wrote: > >>>> > >>>> bq. about having LTS versions once a year > >>>> > >>>> +1 to the above. > >>>> > >>>> There may be various reasons users don't want to upgrade (after new > >>>> releases come out). We should give such users enough flexibility on > the > >>>> upgrade path. > >>>> > >>>> Cheers > >>>> > >>>> On Wed, May 24, 2017 at 8:39 AM, Kostas Kloudas < > >> k.klou...@data-artisans.com > >>>>> wrote: > >>>> > >>>>> Hi all, > >>>>> > >>>>> For the proposal of having a third party tool, I agree with Ted. > >>>>> Maintaining > >>>>> it is a big and far from trivial effort. > >>>>> > >>>>> Now for the window of backwards compatibility, I would argue that > even > >> if > >>>>> for some users 4 months (1 release) is not enough to bump their Flink > >>>>> version, > >>>>> the proposed policy guarantees that there will always be a path from > >> any > >>>>> old > >>>>> version to any subsequent one. > >>>>> > >>>>> Finally, for the proposal about having LTS versions once a year, I am > >> not > >>>>> sure if this will reduce or create more overhead. If I understand the > >> plan > >>>>> correctly, this would mean that the community will have to maintain > >>>>> 2 or 3 LTS versions and the last two major ones, right? > >>>>> > >>>>>> On May 22, 2017, at 7:31 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >>>>>> > >>>>>> For #2, it is difficult to achieve: > >>>>>> > >>>>>> a. maintaining savepoint migration is non-trivial and should be > >> reviewed > >>>>> by > >>>>>> domain experts > >>>>>> b. how to certify such third-party tool > >>>>>> > >>>>>> Cheers > >>>>>> > >>>>>> On Mon, May 22, 2017 at 3:04 AM, 施晓罡 <shixiaoga...@gmail.com> > wrote: > >>>>>> > >>>>>>> Hi all, > >>>>>>> > >>>>>>> Currently, we work a lot in the maintenance of compatibility. > >>>>>>> There exist much code in runtime to support the migration of > >> savepoints > >>>>>>> (most of which are deprecated), making it hard to focus on the > >> current > >>>>>>> implementation. > >>>>>>> When more versions are released, much more efforts will be needed > if > >> we > >>>>>>> try to make these released versions compatible. > >>>>>>> > >>>>>>> I agree with Tzu-Li that we should provide a method to let users > >> upgrade > >>>>>>> Flink in a reasonable pace. > >>>>>>> But i am against the proposal that we only offer backwards > >> compatibility > >>>>>>> for one previous version. > >>>>>>> According our time-based release model, a major version is released > >>>>> every > >>>>>>> four month. > >>>>>>> That means, users have to upgrade their versions every 8 months. > >>>>> Otherwise > >>>>>>> they will have difficulties in the migration of existing > savepoints. > >>>>>>> > >>>>>>> My suggestions include > >>>>>>> > >>>>>>> (1) We can release Long-Term Support (LTS) versions which are > widely > >>>>>>> adopted in other open-source projects. > >>>>>>> LTS versions should be stable and are free of found bugs. > Savepoints > >> in > >>>>>>> LTS versions are guaranteed to be back-compatible so that users can > >>>>> easily > >>>>>>> upgrade to newer LTS versions. > >>>>>>> > >>>>>>> The releasing of LTS versions is slower than that of major versions > >>>>> (maybe > >>>>>>> once a year, determined by users’ upgrade frequency). > >>>>>>> Each LTS version will be supported a period of time and typically > >> there > >>>>>>> are no more than three active LTS versions. > >>>>>>> By encouraging users to use LTS versions, we can ease the > >> maintenance of > >>>>>>> released versions (bug fixes, back compatibility, and critical > >>>>> performance > >>>>>>> improvement). > >>>>>>> > >>>>>>> (2) We can provide a third-party tool to do the migration of > >>>>> old-versioned > >>>>>>> savepoints. > >>>>>>> When users upgrade their versions, they can use the provided tool > to > >>>>>>> migrate existing savepoints. > >>>>>>> This can help move the code for savepoint migration out of the > actual > >>>>>>> codebase, making code focuses on current implementation. > >>>>>>> > >>>>>>> What do you think? > >>>>>>> > >>>>>>> Regards, > >>>>>>> Xiaogang > >>>>>>> > >>>>>>> > >>>>>>>> 在 2017年5月22日，下午1:39，Tzu-Li (Gordon) Tai <tzuli...@apache.org> 写道： > >>>>>>>> > >>>>>>>> Hi Kostas, > >>>>>>>> > >>>>>>>> Thanks for bringing this up! > >>>>>>>> I think it is reasonable to keep this coherent with our > timely-based > >>>>>>> release model guarantees. > >>>>>>>> > >>>>>>>> With the timely-based release model, there is a guarantee that the > >>>>>>> current latest major version and the previous one is supported. > >>>>>>>> For example, upon releasing 1.3, only 1.3 and 1.2 will still be > >>>>>>> supported by the community for any required bug fixes. > >>>>>>>> I think this was initially decided not only to ease old version > >>>>>>> maintenance efforts for the community, but also as a means to let > >> users > >>>>>>> upgrade their Flink versions in a reasonable pace (at least every > >> other > >>>>>>> major release.) > >>>>>>>> > >>>>>>>> Therefore, I think its also reasonable to also clearly state that > >>>>>>> savepoints compatibility will only be guaranteed for the previous > >>>>> release. > >>>>>>>> Although I think at the moment almost, if not all, of the current > >> code > >>>>>>> still maintains compatibility for 1.1, in the long run these > >> migration > >>>>>>> codes would definitely start to pile up and pollute the actual > >> codebase > >>>>> if > >>>>>>> we try to always be compatible with all previous versions. > >>>>>>>> > >>>>>>>> Cheers, > >>>>>>>> Gordon > >>>>>>>> > >>>>>>>> > >>>>>>>> On 21 May 2017 at 2:24:53 AM, Kostas Kloudas ( > >>>>>>> k.klou...@data-artisans.com) wrote: > >>>>>>>> > >>>>>>>> Hi Chesnay, > >>>>>>>> > >>>>>>>> I believe that for APIs we already have a pretty clear policy with > >> the > >>>>>>> annotations. > >>>>>>>> I was referring to savepoints and state related backwards > >>>>> compatibility. > >>>>>>>> > >>>>>>>> > >>>>>>>>> On May 20, 2017, at 7:20 PM, Chesnay Schepler < > ches...@apache.org> > >>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> I think it would be a good to clarify what kind of > >>>>>>> backwards-compatibilitiy we're talking about here. As in are we > >> talking > >>>>>>> about APIs or savepoints? > >>>>>>>>> > >>>>>>>>> On 20.05.2017 19:09, Kostas Kloudas wrote: > >>>>>>>>>> Hi all, > >>>>>>>>>> > >>>>>>>>>> As we are getting closer to releasing Flink-1.3, I would like to > >> open > >>>>>>> a discussion > >>>>>>>>>> on how far back we provide backwards compatibility for. > >>>>>>>>>> > >>>>>>>>>> The reason for opening the discussion is that i) for the users > and > >>>>> for > >>>>>>> the > >>>>>>>>>> adoption of the project, it is good to have an explicitely > stated > >>>>>>> policy that implies > >>>>>>>>>> certain guarantees, and ii) keeping code and tests for backwards > >>>>>>> compatibility with > >>>>>>>>>> Flink-1.1 does not offer much. On the contrary, I think that it > >> leads > >>>>>>> to: > >>>>>>>>>> > >>>>>>>>>> 1) dead or ugly code in the codebase, e.g. deprecated class > fields > >>>>>>> that could go away and > >>>>>>>>>> ugly if() loops (see aligned window operators that were > >> deprecated in > >>>>>>> 1.2 and are now > >>>>>>>>>> normal windows), etc > >>>>>>>>>> 2) expensive tests (as, normally, they read from a savepoint) > >>>>>>>>>> 3) binary files in the codebase for holding the aforementioned > >>>>>>> savepoints > >>>>>>>>>> > >>>>>>>>>> My proposal for such a policy would be to offer backwards > >>>>>>> compatibility for one previous version. > >>>>>>>>>> > >>>>>>>>>> This means that 1.3 will be compatible with 1.2 (not 1.1). This > >> still > >>>>>>> allows a clear > >>>>>>>>>> "backwards compatibility" path when jumping versions (a user > that > >>>>> goes > >>>>>>>>>> from 1.1 to 1.3 can go initially 1.1 -> 1.2, take a savepoint, > and > >>>>>>> then 1.2 -> 1.3), > >>>>>>>>>> while also allowing us to clean up the codebase a bit. > >>>>>>>>>> > >>>>>>>>>> What do you think? > >>>>>>>>>> > >>>>>>>>>> Kostas > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>> > >>>>> > >>> > >> > >> > >
