Hi Thomas, thanks for the feedback and clarification. The Flink community should then make it clear that downstream developers should keep multiple implementations for different Flink versions if it is necessary, which is also a valid concept, so we could focus on the backward compatibility. Hopefully iceberg and hudi developers could join this discussion and confirm it.
Best regards Jing On Wed, Dec 29, 2021 at 8:10 PM Thomas Weise <t...@apache.org> wrote: > Hi Jing, > > AFAIK most of the pain is caused by lack of backward compatibility > (binary). > > And to make sure I'm not adding to the confusion: It would be > necessary to be able to run the iceberg connector built against Flink > 1.12 with a Flink 1.13 distribution. That would solve most problems > downstream projects like Iceberg, Beam etc. face today. Those projects > could then publish their artifacts built against the lowest Flink > version they support and rest assured users with newer versions of > Flink don't run into trouble. > > Thanks, > Thomas > > On Wed, Dec 29, 2021 at 8:04 AM Jing Ge <j...@ververica.com> wrote: > > > > Hi Piotrek, > > > > thanks for asking. To be honest, I hope it could be good enough if Flink > > could only provide backward compatibility, which is easier than providing > > forward compatibility described in the proposal. That is also one of the > > reasons why I started this discussion. If, after the discussion, the > > community could reach a consensus that backward compatibility is fine > > enough, it is also a clear decision for us to avoid any further confusion > > and complaints, i.e. make it clear that there is no guarantee for > > the downstream vendors to provide backward compatibility of their own > > software, they have to maintain implementations with different Flink > minor > > versions simultaneously. In this case, I would move the proposal about > > forward compatibility to the Rejected Alternatives section. > > > > As far as I understood, there are at least three group of users, the > first > > group is the end users who write Flink job/SQL; the second group is > > platform developers who use Flink to build data infrastructure and let > > users in the first group submit their jobs; and the third group is > > downstream software developers(vendors) who build software (module) that > > depends on Flink. Platform developers in the second group will also use > the > > software provided by the third group in their data infra. The developers > > from the second and the third groups play an important role for the > > adaption/upgrade of the new Flink version in the industry. There are > > compatibility issues between them, afaik, both backward and forward. I > was > > not able to join the discussion caused by iceberg upgrade issue > previously, > > after reading the whole story, building the connector with Flink 1.12 > will > > not solve their problem. It was the forward compatibility issue (built > with > > 1.13 and run on 1.12) that made it difficult for them to upgrade. > > > > Best regards > > Jing > > > > On Wed, Dec 29, 2021 at 3:42 PM Piotr Nowojski <pnowoj...@apache.org> > wrote: > > > > > Hi Jink, > > > > > > I haven't yet fully reviewed the FLIP document, but I wanted to clarify > > > something. > > > > > > > Flink Forward Compatibility > > > > Based on the previous clarification, Flink forward compatibility > should > > > mean that Flink jobs or ecosystems like external connectors/formats > built > > > with newer > > > > Flink version Y like 1.15 can be running on an older Flink version X > like > > > 1.14 with no issue. With this definition, we might realize that the > > > original requirement > > > > in the thread [1] was asking for Flink forward compatibility, i.e. > > > iceberg-flink module compiled with Flink 1.13 wanted to be running on > Flink > > > 1.12 cluster. And > > > > from Flink user perspective, this is a backward compatibility issue. > This > > > is the same case: Flink forward compatibility equals Flink > job/ecosystems > > > backward > > > > compatibility. > > > > > > Is it really important to provide this kind of "forward binary" > > > compatibility? You are referring to the iceberg example, but the > workaround > > > there was quite obvious: just build the connector using Flink 1.12. As > long > > > as the connector is using stable, @Public API, AND assuming we provide > > > binary compatibility, it should work fine with Flink 1.13. So to solve > this > > > user request, we could provide either forward OR backward binary > > > compatibility, Right? The question is which one of those is easier to > > > provide and explain to the users. > > > > > > Of course the big missing piece after FLIP-197 is that we haven't > agreed on > > > any type of binary compatibility. > > > > > > Best, > > > Piotrek > > > > > > śr., 29 gru 2021 o 14:07 Jing Ge <j...@ververica.com> napisał(a): > > > > > > > Hi everyone, > > > > > > > > with great interest I have read all discussions [1][2][3] w.r.t. the > > > (API?) > > > > compatibility issues. The feedback coming from the Flink user's > point of > > > > view is very valuable. Many thanks for it. In these discussions, > there > > > were > > > > many explanations that talked about backward and forward > compatibility > > > and > > > > broke down to API and ABI compatibility. Since each time when a > developer > > > > referred to compatibility, there was an implicit context behind, > which > > > > might cause confusion, e.g. the forward compatibility mentioned in > the > > > API > > > > compatibility discussion thread[1] was actually the Flink ABI > backward > > > > compatibility mentioned in FLIP-196. The original requirement posted > in > > > the > > > > API compatibility discussion thread[1] was actually Flink ABI forward > > > > compatibility, afaik. I will explain it in the proposal. They were > all > > > > correct because they talked from different perspectives. But it was > hard > > > > for audiences to follow it. > > > > > > > > I’d like to start a discussion about backward/forward compatibility > from > > > > both users perspective and Flink perspective. I Tried to put myself > in > > > > users' shoes and then to see whether we could do anything to reduce > > > users' > > > > effort for upgrading Flink. > > > > > > > > The proposal contains four parts: > > > > - Clarify the definition of API and ABI, backward and forward > > > compatibility > > > > to make sure everyone is on the same page. > > > > - Analyze and classify issues from users' feedback. > > > > - Summarize the definition and the gap between the current status and > > > what > > > > users need. > > > > - Proposed changes > > > > > > > > Please find the details on > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-207%3A+Flink+backward+and+forward+compatibility > > > > > > > > Looking forward to your feedback. Many thanks. > > > > > > > > best regards > > > > Jing > > > > > > > >
