With the model of externalized Flink connector repo (which I fully support), there is one challenge of supporting versions of two upstream projects (similar to what Peter Vary mentioned earlier).
E.g., today the Flink Iceberg connector lives in Iceberg repo. We have separate modules 1.13, 1.14, 1.15 to list the supported Flink versions, which is still manageable. With the new model, now we may need to multiply that with the number of Iceberg versions that we are going to support, e.g. 0.13, 0.14, 1.0. That multiplication factor would be non-manageable. >From the flink-connector-elasticsearch repo, it is unclear how we define the supported Flink versions. Only one/latest Flink version? On Fri, Sep 30, 2022 at 8:36 AM Péter Váry <peter.vary.apa...@gmail.com> wrote: > +1 having an option storing every version of a connector in one repo > > Also, it would be good to have the major(.minor) version of the connected > system in the name of the connector jar, depending of the compatibility. I > think this compatibility is mostly system dependent. > > Thanks, Peter > > > On Fri, Sep 30, 2022, 09:32 Martijn Visser <martijnvis...@apache.org> > wrote: > > > Hi Peter, > > > > I think this also depends on the support SLA that the technology that you > > connect to provides. For example, with Flink and Elasticsearch, we choose > > to follow Elasticsearch supported versions. So that means that when > support > > for Elasticsearch 8 is introduced, support for Elasticsearch 6 should be > > dropped (since Elastic only support the last major version and the latest > > minor version prior to that) > > > > I don't see value in having different connectors for Iceberg 0.14 and > 0.15 > > in separate repositories. I think that will confuse the user. I would > > expect that with modules you should be able to have support for multiple > > versions in one repository. > > > > Best regards, > > > > Martijn > > > > On Fri, Sep 30, 2022 at 7:44 AM Péter Váry <peter.vary.apa...@gmail.com> > > wrote: > > > > > Thanks for the quick response! > > > > > > Would this mean, that we have different connectors for Iceberg 0.14, > and > > > Iceberg 0.15. Would these different versions kept in different > > repository? > > > > > > My feeling is that this model is fine for the stable/slow moving > systems > > > like Hive/HBase. For other systems, which are evolving faster, this is > > less > > > than ideal. > > > > > > For those, who have more knowledge about the Flink ecosystem: How do > you > > > feel? What is the distribution of the connectors between the slow > moving > > > and the fast moving systems? > > > > > > Thanks, Peter > > > > > > > > > On Thu, Sep 29, 2022, 16:46 Danny Cranmer <dannycran...@apache.org> > > wrote: > > > > > > > If you look at ElasticSearch [1] as an example there are different > > > variants > > > > of the connector depending on the "connected" system: > > > > - flink-connector-elasticsearch6 > > > > - flink-connector-elasticsearch7 > > > > > > > > Looks like Hive and HBase follow a similar pattern in the main Flink > > > repo/ > > > > > > > > [1] https://github.com/apache/flink-connector-elasticsearch > > > > > > > > On Thu, Sep 29, 2022 at 3:17 PM Péter Váry < > > peter.vary.apa...@gmail.com> > > > > wrote: > > > > > > > > > Hi Team, > > > > > > > > > > Just joining the conversation for the first time, so pardon me if I > > > > repeat > > > > > already answered questions. > > > > > > > > > > It might be already discussed, but I think the version for the > > > > "connected" > > > > > system could be important as well. > > > > > > > > > > There might be some API changes between Iceberg 0.14.2, and 1.0.0, > > > which > > > > > would require as to rewrite part of the code for the Flink-Iceberg > > > > > connector. > > > > > It would be important for the users: > > > > > - Which Flink version(s) are this connector working with? > > > > > - Which Iceberg version(s) are this connector working with? > > > > > - Which code version we have for this connector? > > > > > > > > > > Does this make sense? What is the community's experience with the > > > > connected > > > > > systems? Are they stable enough for omitting their version number > > from > > > > the > > > > > naming of the connectors? Would this worth the proliferation of the > > > > > versions? > > > > > > > > > > Thanks, > > > > > Peter > > > > > > > > > > Chesnay Schepler <ches...@apache.org> ezt írta (időpont: 2022. > > szept. > > > > 29., > > > > > Cs, 14:11): > > > > > > > > > > > 2) No; the branch names would not have a Flink version in them; > > > v1.0.0, > > > > > > v1.0.1 etc. > > > > > > > > > > > > On 29/09/2022 14:03, Martijn Visser wrote: > > > > > > > If I summarize it correctly, that means that: > > > > > > > > > > > > > > 1. The versioning scheme would be <major.minor.patch connector > > > > > > > version>-<major.minor supported Flink version>, where there > will > > > > never > > > > > > be a > > > > > > > patch release for a minor version if a newer minor version > > already > > > > > > exists. > > > > > > > E.g., 1.0.0-1.15; 1.0.1-1.15; 1.1.0-1.15; 1.2.0-1.15; > > > > > > > > > > > > > > 2. The branch naming scheme would be > > > > > vmajor.minor-flink-major.flink-minor > > > > > > > E.g., v1.0.0-1.15; v1.0.1-1.15; v1.1.0-1.15; v1.2.0-1.15; > > > > > > > > > > > > > > I would +1 that. > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > Martijn > > > > > > > > > > > > > > On Tue, Sep 20, 2022 at 2:21 PM Chesnay Schepler < > > > ches...@apache.org > > > > > > > > > > > wrote: > > > > > > > > > > > > > >> > After 1.16, only patches are accepted for 1.2.0-1.15. > > > > > > >> > > > > > > >> I feel like this is a misunderstanding that both you and Danny > > ran > > > > > into. > > > > > > >> > > > > > > >> What I meant in the original proposal is that the last 2 > _major_ > > > > > > >> /connector /versions are supported, with the latest receiving > > > > > additional > > > > > > >> features. > > > > > > >> (Provided that the previous major version still works against > a > > > > > > >> currently supported Flink version!) > > > > > > >> There will never be patch releases for a minor version if a > > newer > > > > > minor > > > > > > >> version exists. > > > > > > >> > > > > > > >> IOW, the minor/patch releases within a major version do not > > form a > > > > > tree > > > > > > >> (like in Flink), but a line. > > > > > > >> > > > > > > >> 1.0.0 -> 1.0.1 -> 1.1.0 -> 1.2.0 -> ... > > > > > > >> NOT > > > > > > >> 1.0.0 -> 1.0.1 -> 1.0.2 > > > > > > >> |-----> 1.1.0 -> 1.1.1 > > > > > > >> > > > > > > >> If we actually follow semantic versioning then it's just not > > > > necessary > > > > > > >> to publish a patch for a previous version. > > > > > > >> > > > > > > >> So if 2.x exists, then (the latest) 2.x gets features and > > patches, > > > > and > > > > > > >> the latest 1.x gets patches. > > > > > > >> > > > > > > >> I hope that clears things up. > > > > > > >> > > > > > > >> On 20/09/2022 14:00, Jing Ge wrote: > > > > > > >>> Hi, > > > > > > >>> > > > > > > >>> Thanks for starting this discussion. It is an interesting one > > and > > > > > yeah, > > > > > > >> it > > > > > > >>> is a tough topic. It seems like a centralized release version > > > > schema > > > > > > >>> control for decentralized connector development ;-) > > > > > > >>> > > > > > > >>> In general, I like this idea, not because it is a good one > but > > > > > because > > > > > > >>> there might be no better one(That's life!). The solution > gives > > > > users > > > > > an > > > > > > >>> easy life with the price of extra effort on the developer's > > part. > > > > But > > > > > > it > > > > > > >> is > > > > > > >>> a chicken and egg situation, i.e. developer friendly vs. user > > > > > friendly. > > > > > > >> If > > > > > > >>> it is hard for developers to move forward, it will also be > > > > difficult > > > > > > for > > > > > > >>> users to get a new release, even if the version schema is > user > > > > > > friendly. > > > > > > >>> > > > > > > >>> I'd like to raise some questions/concerns to make sure we are > > on > > > > the > > > > > > same > > > > > > >>> page. > > > > > > >>> > > > > > > >>> @Chesnay > > > > > > >>> > > > > > > >>> c1) Imagine we have 2.0.0 for 1.15: > > > > > > >>> > > > > > > >>> - 2.0.0-1.14 (patches) > > > > > > >>> - 2.0.0-1.15 (feature and patches) > > > > > > >>> ===> new major release targeting 1.16 and we need to change > > code > > > > for > > > > > > new > > > > > > >> API > > > > > > >>> - 2.0.0-1.14 (no support) > > > > > > >>> - 2.0.0-1.15 (patches) > > > > > > >>> - 2.0.1-1.15 (new patches) > > > > > > >>> - 2.1.0-1.16 (feature and patches) > > > > > > >>> > > > > > > >>> There is no more 2.1.0-1.15 because only the latest version > is > > > > > > receiving > > > > > > >>> new features. > > > > > > >>> > > > > > > >>> b1) Even if in some special cases that we need to break the > > rule, > > > > we > > > > > > >> should > > > > > > >>> avoid confusing users: > > > > > > >>> > > > > > > >>> ===> new major release targeting 1.16 and we need to change > > code > > > > for > > > > > > new > > > > > > >> API > > > > > > >>> - 2.0.0-1.14 (no support) > > > > > > >>> - 2.0.0-1.15 (patches) > > > > > > >>> - 2.1.0-1.16 (feature and patches) > > > > > > >>> ===> now we want to break the rule to add features to the > > > > penultimate > > > > > > >>> version > > > > > > >>> - 2.0.0-1.14 (no support) > > > > > > >>> - 2.0.0-1.15 (patches) > > > > > > >>> - 2.2.0-1.15 (patches, new features) // 2.1.0-1.15 vs. > > > > > > 2.2.0-1.15, > > > > > > >>> have to choose 2.2.0-1.15 to avoid conflict > > > > > > >>> - 2.1.0-1.16 (feature and patches) > > > > > > >>> > > > > > > >>> we have two options: 2.1.0-1.15 vs. 2.2.0-1.15, both will > > confuse > > > > > > users: > > > > > > >>> - Using 2.1.0-1.15 will conflict with the existing > 2.1.0-1.16. > > > The > > > > > > >>> connector version of "2.1.0-1.16" is actually 2.1.0 which > means > > > it > > > > > has > > > > > > >> the > > > > > > >>> same code as 2.1.0-1.15 but in this case, it contains > upgraded > > > > code. > > > > > > >>> - Using 2.2.0-1.15 will skip 2.1.0-1.15. Actually, it needs > to > > > skip > > > > > all > > > > > > >>> occupied minor-1.16 versions, heads-up release manager! > > > > > > >>> > > > > > > >>> c2) Allow me using your example: > > > > > > >>> > > > > > > >>> ===> new major release targeting 1.16 > > > > > > >>> - 1.2.0-1.14 (no support; unsupported Flink version) > > > > > > >>> - 1.2.0-1.15 (patches; supported until either 3.0 of 1.17, > > > > whichever > > > > > > >>> happens first) > > > > > > >>> - 2.0.0-1.15 (feature and patches) > > > > > > >>> - 2.0.0-1.16 (feature and patches) > > > > > > >>> > > > > > > >>> I didn't understand the part of "2.0.0-1.15 (features and > > > > patches)". > > > > > > >> After > > > > > > >>> 1.16, only patches are accepted for 1.2.0-1.15. > > > > > > >>> It should be clearly defined how to bump up the connector's > > > version > > > > > > >> number > > > > > > >>> for the new Flink version. If the connector major number > would > > > > always > > > > > > >> bump > > > > > > >>> up, it would make less sense to use the Flink version as > > postfix. > > > > > With > > > > > > >> the > > > > > > >>> same example, it should be: > > > > > > >>> > > > > > > >>> ===> new major release targeting 1.16 > > > > > > >>> - 1.2.0-1.14 (no support; unsupported Flink version) > > > > > > >>> - 1.2.0-1.15 (patches; supported until either 3.0 of 1.17, > > > > whichever > > > > > > >>> happens first) > > > > > > >>> - 1.2.1-1.15 (new patches) > > > > > > >>> - 1.3.0-1.16 (feature and patches) > > > > > > >>> - 1.4.0-1.16 (feature and patches, new features) > > > > > > >>> - 2.0.0-1.16 (feature and patches, major upgrade of > > > connector > > > > > > >> itself) > > > > > > >>> or > > > > > > >>> > > > > > > >>> - 1.2.0-1.14 (patches) > > > > > > >>> - 1.2.0-1.15 (feature and patches) > > > > > > >>> - 2.0.0 -1.15 (feature and patches, major upgrade of > > > > connector > > > > > > >> itself) > > > > > > >>> ===> new major release targeting 1.16 > > > > > > >>> - 1.2.0-1.14 (no support; unsupported Flink version) > > > > > > >>> - 2.0.0-1.15 (patches) > > > > > > >>> - 2.0.1-1.15 (new patches) > > > > > > >>> - 2.1.0-1.16 (feature and patches) > > > > > > >>> - 2.2.0-1.16 (feature and patches, new features) > > > > > > >>> > > > > > > >>> i.e. commonly, there should be no connector major version > > change > > > > when > > > > > > >> using > > > > > > >>> the Flink version postfix as the version schema. Special > > > > cases(rarely > > > > > > >>> happens) are obviously allowed. > > > > > > >>> > > > > > > >>> Best regards, > > > > > > >>> Jing > > > > > > >>> > > > > > > >>> On Tue, Sep 20, 2022 at 10:57 AM Martijn Visser< > > > > > > martijnvis...@apache.org > > > > > > >>> > > > > > > >>> wrote: > > > > > > >>> > > > > > > >>>> Hi all, > > > > > > >>>> > > > > > > >>>> This is a tough topic, I also had to write things down a > > couple > > > of > > > > > > >> times. > > > > > > >>>> To summarize and add my thoughts: > > > > > > >>>> > > > > > > >>>> a) I think everyone is agreeing that "Only the last 2 > versions > > > of > > > > a > > > > > > >>>> connector are supported per supported Flink version, with > only > > > the > > > > > > >> latest > > > > > > >>>> version receiving new features". In the current situation, > > that > > > > > means > > > > > > >> that > > > > > > >>>> Flink 1.14 and Flink 1.15 would be supported for connectors. > > > This > > > > > > >> results > > > > > > >>>> in a maximum of 4 supported connector versions. > > > > > > >>>> > > > > > > >>>> b1) In an ideal world, I would have liked Flink's APIs that > > are > > > > used > > > > > > by > > > > > > >>>> connectors to be versioned (that's why there's now a Sink V1 > > > and a > > > > > > Sink > > > > > > >>>> V2). However, we're not there yet. > > > > > > >>>> > > > > > > >>>> b2) With regards to the remark of using @Interal APIs, one > > thing > > > > > that > > > > > > we > > > > > > >>>> agreed to in previous discussions is that connectors > shouldn't > > > > need > > > > > to > > > > > > >> rely > > > > > > >>>> on @Interal APIs so that the connector surface also > > stabilizes. > > > > > > >>>> > > > > > > >>>> b3) In the end, I think what matters the most is the user's > > > > > perception > > > > > > >> on > > > > > > >>>> versioning. So the first thing to establish would be the > > > > versioning > > > > > > for > > > > > > >>>> connectors itself. So you would indeed have a > > > <major.minor.patch> > > > > > > >> scheme. > > > > > > >>>> Next is the compatibility of that scheme with a version of > > > Flink. > > > > I > > > > > do > > > > > > >> like > > > > > > >>>> Chesnay's approach for using the Scala suffixes idea. So you > > > would > > > > > > have > > > > > > >>>> <major.minor.patch connector>_<major.minor Flink version>. > In > > > the > > > > > > >> currently > > > > > > >>>> externalized Elasticsearch connector, we would end up with > > > > > 3.0.0_1.14 > > > > > > >> and > > > > > > >>>> 3.0.0_1.15 as first released versions. If a new Flink > version > > > > would > > > > > be > > > > > > >>>> released that doesn't require code changes to the connector, > > the > > > > > > >> released > > > > > > >>>> version would be 3.0.0_1.16. That means that there have been > > no > > > > > > >> connector > > > > > > >>>> code changes (no patches, no new features) when comparing > this > > > > > across > > > > > > >>>> different Flink versions. > > > > > > >>>> > > > > > > >>>> b4) Now using the example that Chesnay provided (yet > slightly > > > > > modified > > > > > > >> to > > > > > > >>>> match it with the Elasticsearch example I've used above), > > there > > > > > exists > > > > > > >> an > > > > > > >>>> Elasticsearch connector 3.0.0_1.15. Now in Flink 1.16, > > there's a > > > > new > > > > > > API > > > > > > >>>> that we want to use, which is a test util. It would result > in > > > > > version > > > > > > >>>> 3.1.0_1.16 for the new Flink version. Like Chesnay said, for > > the > > > > > sake > > > > > > of > > > > > > >>>> argument, at the same time we also had some pending changes > > for > > > > the > > > > > > 1.15 > > > > > > >>>> connector (let's say exclusive to 1.15; some workaround for > a > > > bug > > > > or > > > > > > >> smth), > > > > > > >>>> so we would also end up with 3.1.0-1.15. I agree with Danny > > that > > > > we > > > > > > >> should > > > > > > >>>> avoid this situation: the perception of the user would be > that > > > > > there's > > > > > > >> no > > > > > > >>>> divergence between the 3.1.0 version, except the compatible > > > Flink > > > > > > >> version. > > > > > > >>>> I really am wondering how often we will run in that > situation. > > > > From > > > > > > what > > > > > > >>>> I've seen so far with connectors is that bug fixes always > end > > up > > > > in > > > > > > both > > > > > > >>>> the release branch and the master branch. The only > exceptions > > > are > > > > > test > > > > > > >>>> stabilities or documentation fixes, but if we only resolve > > > these, > > > > > they > > > > > > >>>> wouldn't need to be released. If such a special occasion > would > > > > > occur, > > > > > > I > > > > > > >>>> would be inclined to go for a hotfix approach, where you > would > > > end > > > > > up > > > > > > >> with > > > > > > >>>> 3.0.0.1_1.15. > > > > > > >>>> > > > > > > >>>> c) Branch wise, I think we should end up with > > <major.minor.patch > > > > > > >>>> connector>_<major.minor Flink version>. So again the > > > Elasticsearch > > > > > > >> example, > > > > > > >>>> at this moment there would be 3.0.0_1.14 and 3.0.0_1.15 > > > branches. > > > > > > >>>> > > > > > > >>>> Best regards, > > > > > > >>>> > > > > > > >>>> Martijn > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
