I agree that 1.1 compatibility is the most important “pain point", as 
compatibility with the rest of the versions follows a more “systematic” 
approach.

I think that discarding compatibility with 1.1 will clear some parts 
of the codebase significantly.

Kostas

> On Jun 27, 2017, at 6:03 PM, Stephan Ewen <se...@apache.org> wrote:
> 
> I think that this discussion is probably motivated especially by the
> "legacy state" handling of Flink 1.1.
> The biggest gain in codebase and productivity would be won only by dropping
> 1.1 compatibility in Flink 1.4.
> 
> My gut feeling is that this is reasonable. We support two versions back,
> which means that users can skip one upgrade, but not two.
> 
> From what I can tell, users are usually eager to upgrade. They don't do it
> immediately, but as soon as the new release is a bit battle tested.
> 
> I would expect skipping two entire versions to be rare enough to be okay
> with a solution which is a bit more effort for the user:
> You can upgrade from Flink 1.1. to 1.4 by loading the 1.1 savepoint into
> Flink 1.2, take a savepoint (1.2 format), and resume that in Flink 1.4.
> 
> Greetings,
> Stephan
> 
> 
> On Tue, Jun 27, 2017 at 12:01 PM, Stefan Richter <
> s.rich...@data-artisans.com> wrote:
> 
>> For many parts of the code, I would agree with Aljoscha. However, I can
>> also see notable exceptions, such as maintaining support for the legacy
>> state from Flink <=1.1. For example, I think dropping support for this can
>> simplify new developments such as fast local recovery or state replication
>> quiet a bit because this is a special case that runs through a lot of code
>> from backend to JM. So besides this general discussion about a backwards
>> compatible policy, do you think it could make sense to start another
>> concrete discussion about if we still must or want backwards compatibility
>> to Flink 1.1 in Flink 1.4?
>> 
>>> Am 29.05.2017 um 12:08 schrieb Aljoscha Krettek <aljos...@apache.org>:
>>> 
>>> Normally, I’m the first one to suggest removing everything that is not
>> absolutely necessary in order to have a clean code base. On this issue,
>> though, I think we should support restoring from old Savepoints as far back
>> as possible if it does not make the code completely unmaintainable. Some
>> users might jump versions and always forcing them to go though every
>> version from their old version to the current version doesn’t seem feasible
>> and might put off some users.
>>> 
>>> So far, I think the burden of supporting restore from 1.1 is still small
>> enough and with each new version the changes between versions become less
>> and less. The changes from 1.2 to the upcoming 1.3 are quite minimal, I
>> think.
>>> 
>>> Best,
>>> Aljoscha
>>>> On 24. May 2017, at 17:58, Ted Yu <yuzhih...@gmail.com> wrote:
>>>> 
>>>> bq. about having LTS versions once a year
>>>> 
>>>> +1 to the above.
>>>> 
>>>> There may be various reasons users don't want to upgrade (after new
>>>> releases come out). We should give such users enough flexibility on the
>>>> upgrade path.
>>>> 
>>>> Cheers
>>>> 
>>>> On Wed, May 24, 2017 at 8:39 AM, Kostas Kloudas <
>> k.klou...@data-artisans.com
>>>>> wrote:
>>>> 
>>>>> Hi all,
>>>>> 
>>>>> For the proposal of having a third party tool, I agree with Ted.
>>>>> Maintaining
>>>>> it is a big and far from trivial effort.
>>>>> 
>>>>> Now for the window of backwards compatibility, I would argue that even
>> if
>>>>> for some users 4 months (1 release) is not enough to bump their Flink
>>>>> version,
>>>>> the proposed policy guarantees that there will always be a path from
>> any
>>>>> old
>>>>> version to any subsequent one.
>>>>> 
>>>>> Finally, for the proposal about having LTS versions once a year, I am
>> not
>>>>> sure if this will reduce or create more overhead. If I understand the
>> plan
>>>>> correctly, this would mean that the community will have to maintain
>>>>> 2 or 3 LTS versions and the last two major ones, right?
>>>>> 
>>>>>> On May 22, 2017, at 7:31 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>>>>>> 
>>>>>> For #2, it is difficult to achieve:
>>>>>> 
>>>>>> a. maintaining savepoint migration is non-trivial and should be
>> reviewed
>>>>> by
>>>>>> domain experts
>>>>>> b. how to certify such third-party tool
>>>>>> 
>>>>>> Cheers
>>>>>> 
>>>>>> On Mon, May 22, 2017 at 3:04 AM, 施晓罡 <shixiaoga...@gmail.com> wrote:
>>>>>> 
>>>>>>> Hi all,
>>>>>>> 
>>>>>>> Currently, we work a lot in the maintenance of compatibility.
>>>>>>> There exist much code in runtime to support the migration of
>> savepoints
>>>>>>> (most of which are deprecated), making it hard to focus on the
>> current
>>>>>>> implementation.
>>>>>>> When more versions are released, much more efforts will be needed if
>> we
>>>>>>> try to make these released versions compatible.
>>>>>>> 
>>>>>>> I agree with Tzu-Li that we should provide a method to let users
>> upgrade
>>>>>>> Flink in a reasonable pace.
>>>>>>> But i am against the proposal that we only offer backwards
>> compatibility
>>>>>>> for one previous version.
>>>>>>> According our time-based release model, a major version is released
>>>>> every
>>>>>>> four month.
>>>>>>> That means, users have to upgrade their versions every 8 months.
>>>>> Otherwise
>>>>>>> they will have difficulties in the migration of existing savepoints.
>>>>>>> 
>>>>>>> My suggestions include
>>>>>>> 
>>>>>>> (1) We can release Long-Term Support (LTS) versions which are widely
>>>>>>> adopted in other open-source projects.
>>>>>>> LTS versions should be stable and are free of found bugs. Savepoints
>> in
>>>>>>> LTS versions are guaranteed to be back-compatible so that users can
>>>>> easily
>>>>>>> upgrade to newer LTS versions.
>>>>>>> 
>>>>>>> The releasing of LTS versions is slower than that of major versions
>>>>> (maybe
>>>>>>> once a year, determined by users’ upgrade frequency).
>>>>>>> Each LTS version will be supported a period of time and typically
>> there
>>>>>>> are no more than three active LTS versions.
>>>>>>> By encouraging users to use LTS versions, we can ease the
>> maintenance of
>>>>>>> released versions (bug fixes, back compatibility, and critical
>>>>> performance
>>>>>>> improvement).
>>>>>>> 
>>>>>>> (2) We can provide a third-party tool to do the migration of
>>>>> old-versioned
>>>>>>> savepoints.
>>>>>>> When users upgrade their versions, they can use the provided tool to
>>>>>>> migrate existing savepoints.
>>>>>>> This can help move the code for savepoint migration out of the actual
>>>>>>> codebase,  making code focuses on current implementation.
>>>>>>> 
>>>>>>> What do you think?
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Xiaogang
>>>>>>> 
>>>>>>> 
>>>>>>>> 在 2017年5月22日,下午1:39,Tzu-Li (Gordon) Tai <tzuli...@apache.org> 写道:
>>>>>>>> 
>>>>>>>> Hi Kostas,
>>>>>>>> 
>>>>>>>> Thanks for bringing this up!
>>>>>>>> I think it is reasonable to keep this coherent with our timely-based
>>>>>>> release model guarantees.
>>>>>>>> 
>>>>>>>> With the timely-based release model, there is a guarantee that the
>>>>>>> current latest major version and the previous one is supported.
>>>>>>>> For example, upon releasing 1.3, only 1.3 and 1.2 will still be
>>>>>>> supported by the community for any required bug fixes.
>>>>>>>> I think this was initially decided not only to ease old version
>>>>>>> maintenance efforts for the community, but also as a means to let
>> users
>>>>>>> upgrade their Flink versions in a reasonable pace (at least every
>> other
>>>>>>> major release.)
>>>>>>>> 
>>>>>>>> Therefore, I think its also reasonable to also clearly state that
>>>>>>> savepoints compatibility will only be guaranteed for the previous
>>>>> release.
>>>>>>>> Although I think at the moment almost, if not all, of the current
>> code
>>>>>>> still maintains compatibility for 1.1, in the long run these
>> migration
>>>>>>> codes would definitely start to pile up and pollute the actual
>> codebase
>>>>> if
>>>>>>> we try to always be compatible with all previous versions.
>>>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> Gordon
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 21 May 2017 at 2:24:53 AM, Kostas Kloudas (
>>>>>>> k.klou...@data-artisans.com) wrote:
>>>>>>>> 
>>>>>>>> Hi Chesnay,
>>>>>>>> 
>>>>>>>> I believe that for APIs we already have a pretty clear policy with
>> the
>>>>>>> annotations.
>>>>>>>> I was referring to savepoints and state related backwards
>>>>> compatibility.
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On May 20, 2017, at 7:20 PM, Chesnay Schepler <ches...@apache.org>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> I think it would be a good to clarify what kind of
>>>>>>> backwards-compatibilitiy we're talking about here. As in are we
>> talking
>>>>>>> about APIs or savepoints?
>>>>>>>>> 
>>>>>>>>> On 20.05.2017 19:09, Kostas Kloudas wrote:
>>>>>>>>>> Hi all,
>>>>>>>>>> 
>>>>>>>>>> As we are getting closer to releasing Flink-1.3, I would like to
>> open
>>>>>>> a discussion
>>>>>>>>>> on how far back we provide backwards compatibility for.
>>>>>>>>>> 
>>>>>>>>>> The reason for opening the discussion is that i) for the users and
>>>>> for
>>>>>>> the
>>>>>>>>>> adoption of the project, it is good to have an explicitely stated
>>>>>>> policy that implies
>>>>>>>>>> certain guarantees, and ii) keeping code and tests for backwards
>>>>>>> compatibility with
>>>>>>>>>> Flink-1.1 does not offer much. On the contrary, I think that it
>> leads
>>>>>>> to:
>>>>>>>>>> 
>>>>>>>>>> 1) dead or ugly code in the codebase, e.g. deprecated class fields
>>>>>>> that could go away and
>>>>>>>>>> ugly if() loops (see aligned window operators that were
>> deprecated in
>>>>>>> 1.2 and are now
>>>>>>>>>> normal windows), etc
>>>>>>>>>> 2) expensive tests (as, normally, they read from a savepoint)
>>>>>>>>>> 3) binary files in the codebase for holding the aforementioned
>>>>>>> savepoints
>>>>>>>>>> 
>>>>>>>>>> My proposal for such a policy would be to offer backwards
>>>>>>> compatibility for one previous version.
>>>>>>>>>> 
>>>>>>>>>> This means that 1.3 will be compatible with 1.2 (not 1.1). This
>> still
>>>>>>> allows a clear
>>>>>>>>>> "backwards compatibility" path when jumping versions (a user that
>>>>> goes
>>>>>>>>>> from 1.1 to 1.3 can go initially 1.1 -> 1.2, take a savepoint, and
>>>>>>> then 1.2 -> 1.3),
>>>>>>>>>> while also allowing us to clean up the codebase a bit.
>>>>>>>>>> 
>>>>>>>>>> What do you think?
>>>>>>>>>> 
>>>>>>>>>> Kostas
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>> 
>> 
>> 

Reply via email to