+1 having an option storing every version of a connector in one repo Also, it would be good to have the major(.minor) version of the connected system in the name of the connector jar, depending of the compatibility. I think this compatibility is mostly system dependent.
Thanks, Peter On Fri, Sep 30, 2022, 09:32 Martijn Visser <martijnvis...@apache.org> wrote: > Hi Peter, > > I think this also depends on the support SLA that the technology that you > connect to provides. For example, with Flink and Elasticsearch, we choose > to follow Elasticsearch supported versions. So that means that when support > for Elasticsearch 8 is introduced, support for Elasticsearch 6 should be > dropped (since Elastic only support the last major version and the latest > minor version prior to that) > > I don't see value in having different connectors for Iceberg 0.14 and 0.15 > in separate repositories. I think that will confuse the user. I would > expect that with modules you should be able to have support for multiple > versions in one repository. > > Best regards, > > Martijn > > On Fri, Sep 30, 2022 at 7:44 AM Péter Váry <peter.vary.apa...@gmail.com> > wrote: > > > Thanks for the quick response! > > > > Would this mean, that we have different connectors for Iceberg 0.14, and > > Iceberg 0.15. Would these different versions kept in different > repository? > > > > My feeling is that this model is fine for the stable/slow moving systems > > like Hive/HBase. For other systems, which are evolving faster, this is > less > > than ideal. > > > > For those, who have more knowledge about the Flink ecosystem: How do you > > feel? What is the distribution of the connectors between the slow moving > > and the fast moving systems? > > > > Thanks, Peter > > > > > > On Thu, Sep 29, 2022, 16:46 Danny Cranmer <dannycran...@apache.org> > wrote: > > > > > If you look at ElasticSearch [1] as an example there are different > > variants > > > of the connector depending on the "connected" system: > > > - flink-connector-elasticsearch6 > > > - flink-connector-elasticsearch7 > > > > > > Looks like Hive and HBase follow a similar pattern in the main Flink > > repo/ > > > > > > [1] https://github.com/apache/flink-connector-elasticsearch > > > > > > On Thu, Sep 29, 2022 at 3:17 PM Péter Váry < > peter.vary.apa...@gmail.com> > > > wrote: > > > > > > > Hi Team, > > > > > > > > Just joining the conversation for the first time, so pardon me if I > > > repeat > > > > already answered questions. > > > > > > > > It might be already discussed, but I think the version for the > > > "connected" > > > > system could be important as well. > > > > > > > > There might be some API changes between Iceberg 0.14.2, and 1.0.0, > > which > > > > would require as to rewrite part of the code for the Flink-Iceberg > > > > connector. > > > > It would be important for the users: > > > > - Which Flink version(s) are this connector working with? > > > > - Which Iceberg version(s) are this connector working with? > > > > - Which code version we have for this connector? > > > > > > > > Does this make sense? What is the community's experience with the > > > connected > > > > systems? Are they stable enough for omitting their version number > from > > > the > > > > naming of the connectors? Would this worth the proliferation of the > > > > versions? > > > > > > > > Thanks, > > > > Peter > > > > > > > > Chesnay Schepler <ches...@apache.org> ezt írta (időpont: 2022. > szept. > > > 29., > > > > Cs, 14:11): > > > > > > > > > 2) No; the branch names would not have a Flink version in them; > > v1.0.0, > > > > > v1.0.1 etc. > > > > > > > > > > On 29/09/2022 14:03, Martijn Visser wrote: > > > > > > If I summarize it correctly, that means that: > > > > > > > > > > > > 1. The versioning scheme would be <major.minor.patch connector > > > > > > version>-<major.minor supported Flink version>, where there will > > > never > > > > > be a > > > > > > patch release for a minor version if a newer minor version > already > > > > > exists. > > > > > > E.g., 1.0.0-1.15; 1.0.1-1.15; 1.1.0-1.15; 1.2.0-1.15; > > > > > > > > > > > > 2. The branch naming scheme would be > > > > vmajor.minor-flink-major.flink-minor > > > > > > E.g., v1.0.0-1.15; v1.0.1-1.15; v1.1.0-1.15; v1.2.0-1.15; > > > > > > > > > > > > I would +1 that. > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Martijn > > > > > > > > > > > > On Tue, Sep 20, 2022 at 2:21 PM Chesnay Schepler < > > ches...@apache.org > > > > > > > > > wrote: > > > > > > > > > > > >> > After 1.16, only patches are accepted for 1.2.0-1.15. > > > > > >> > > > > > >> I feel like this is a misunderstanding that both you and Danny > ran > > > > into. > > > > > >> > > > > > >> What I meant in the original proposal is that the last 2 _major_ > > > > > >> /connector /versions are supported, with the latest receiving > > > > additional > > > > > >> features. > > > > > >> (Provided that the previous major version still works against a > > > > > >> currently supported Flink version!) > > > > > >> There will never be patch releases for a minor version if a > newer > > > > minor > > > > > >> version exists. > > > > > >> > > > > > >> IOW, the minor/patch releases within a major version do not > form a > > > > tree > > > > > >> (like in Flink), but a line. > > > > > >> > > > > > >> 1.0.0 -> 1.0.1 -> 1.1.0 -> 1.2.0 -> ... > > > > > >> NOT > > > > > >> 1.0.0 -> 1.0.1 -> 1.0.2 > > > > > >> |-----> 1.1.0 -> 1.1.1 > > > > > >> > > > > > >> If we actually follow semantic versioning then it's just not > > > necessary > > > > > >> to publish a patch for a previous version. > > > > > >> > > > > > >> So if 2.x exists, then (the latest) 2.x gets features and > patches, > > > and > > > > > >> the latest 1.x gets patches. > > > > > >> > > > > > >> I hope that clears things up. > > > > > >> > > > > > >> On 20/09/2022 14:00, Jing Ge wrote: > > > > > >>> Hi, > > > > > >>> > > > > > >>> Thanks for starting this discussion. It is an interesting one > and > > > > yeah, > > > > > >> it > > > > > >>> is a tough topic. It seems like a centralized release version > > > schema > > > > > >>> control for decentralized connector development ;-) > > > > > >>> > > > > > >>> In general, I like this idea, not because it is a good one but > > > > because > > > > > >>> there might be no better one(That's life!). The solution gives > > > users > > > > an > > > > > >>> easy life with the price of extra effort on the developer's > part. > > > But > > > > > it > > > > > >> is > > > > > >>> a chicken and egg situation, i.e. developer friendly vs. user > > > > friendly. > > > > > >> If > > > > > >>> it is hard for developers to move forward, it will also be > > > difficult > > > > > for > > > > > >>> users to get a new release, even if the version schema is user > > > > > friendly. > > > > > >>> > > > > > >>> I'd like to raise some questions/concerns to make sure we are > on > > > the > > > > > same > > > > > >>> page. > > > > > >>> > > > > > >>> @Chesnay > > > > > >>> > > > > > >>> c1) Imagine we have 2.0.0 for 1.15: > > > > > >>> > > > > > >>> - 2.0.0-1.14 (patches) > > > > > >>> - 2.0.0-1.15 (feature and patches) > > > > > >>> ===> new major release targeting 1.16 and we need to change > code > > > for > > > > > new > > > > > >> API > > > > > >>> - 2.0.0-1.14 (no support) > > > > > >>> - 2.0.0-1.15 (patches) > > > > > >>> - 2.0.1-1.15 (new patches) > > > > > >>> - 2.1.0-1.16 (feature and patches) > > > > > >>> > > > > > >>> There is no more 2.1.0-1.15 because only the latest version is > > > > > receiving > > > > > >>> new features. > > > > > >>> > > > > > >>> b1) Even if in some special cases that we need to break the > rule, > > > we > > > > > >> should > > > > > >>> avoid confusing users: > > > > > >>> > > > > > >>> ===> new major release targeting 1.16 and we need to change > code > > > for > > > > > new > > > > > >> API > > > > > >>> - 2.0.0-1.14 (no support) > > > > > >>> - 2.0.0-1.15 (patches) > > > > > >>> - 2.1.0-1.16 (feature and patches) > > > > > >>> ===> now we want to break the rule to add features to the > > > penultimate > > > > > >>> version > > > > > >>> - 2.0.0-1.14 (no support) > > > > > >>> - 2.0.0-1.15 (patches) > > > > > >>> - 2.2.0-1.15 (patches, new features) // 2.1.0-1.15 vs. > > > > > 2.2.0-1.15, > > > > > >>> have to choose 2.2.0-1.15 to avoid conflict > > > > > >>> - 2.1.0-1.16 (feature and patches) > > > > > >>> > > > > > >>> we have two options: 2.1.0-1.15 vs. 2.2.0-1.15, both will > confuse > > > > > users: > > > > > >>> - Using 2.1.0-1.15 will conflict with the existing 2.1.0-1.16. > > The > > > > > >>> connector version of "2.1.0-1.16" is actually 2.1.0 which means > > it > > > > has > > > > > >> the > > > > > >>> same code as 2.1.0-1.15 but in this case, it contains upgraded > > > code. > > > > > >>> - Using 2.2.0-1.15 will skip 2.1.0-1.15. Actually, it needs to > > skip > > > > all > > > > > >>> occupied minor-1.16 versions, heads-up release manager! > > > > > >>> > > > > > >>> c2) Allow me using your example: > > > > > >>> > > > > > >>> ===> new major release targeting 1.16 > > > > > >>> - 1.2.0-1.14 (no support; unsupported Flink version) > > > > > >>> - 1.2.0-1.15 (patches; supported until either 3.0 of 1.17, > > > whichever > > > > > >>> happens first) > > > > > >>> - 2.0.0-1.15 (feature and patches) > > > > > >>> - 2.0.0-1.16 (feature and patches) > > > > > >>> > > > > > >>> I didn't understand the part of "2.0.0-1.15 (features and > > > patches)". > > > > > >> After > > > > > >>> 1.16, only patches are accepted for 1.2.0-1.15. > > > > > >>> It should be clearly defined how to bump up the connector's > > version > > > > > >> number > > > > > >>> for the new Flink version. If the connector major number would > > > always > > > > > >> bump > > > > > >>> up, it would make less sense to use the Flink version as > postfix. > > > > With > > > > > >> the > > > > > >>> same example, it should be: > > > > > >>> > > > > > >>> ===> new major release targeting 1.16 > > > > > >>> - 1.2.0-1.14 (no support; unsupported Flink version) > > > > > >>> - 1.2.0-1.15 (patches; supported until either 3.0 of 1.17, > > > whichever > > > > > >>> happens first) > > > > > >>> - 1.2.1-1.15 (new patches) > > > > > >>> - 1.3.0-1.16 (feature and patches) > > > > > >>> - 1.4.0-1.16 (feature and patches, new features) > > > > > >>> - 2.0.0-1.16 (feature and patches, major upgrade of > > connector > > > > > >> itself) > > > > > >>> or > > > > > >>> > > > > > >>> - 1.2.0-1.14 (patches) > > > > > >>> - 1.2.0-1.15 (feature and patches) > > > > > >>> - 2.0.0 -1.15 (feature and patches, major upgrade of > > > connector > > > > > >> itself) > > > > > >>> ===> new major release targeting 1.16 > > > > > >>> - 1.2.0-1.14 (no support; unsupported Flink version) > > > > > >>> - 2.0.0-1.15 (patches) > > > > > >>> - 2.0.1-1.15 (new patches) > > > > > >>> - 2.1.0-1.16 (feature and patches) > > > > > >>> - 2.2.0-1.16 (feature and patches, new features) > > > > > >>> > > > > > >>> i.e. commonly, there should be no connector major version > change > > > when > > > > > >> using > > > > > >>> the Flink version postfix as the version schema. Special > > > cases(rarely > > > > > >>> happens) are obviously allowed. > > > > > >>> > > > > > >>> Best regards, > > > > > >>> Jing > > > > > >>> > > > > > >>> On Tue, Sep 20, 2022 at 10:57 AM Martijn Visser< > > > > > martijnvis...@apache.org > > > > > >>> > > > > > >>> wrote: > > > > > >>> > > > > > >>>> Hi all, > > > > > >>>> > > > > > >>>> This is a tough topic, I also had to write things down a > couple > > of > > > > > >> times. > > > > > >>>> To summarize and add my thoughts: > > > > > >>>> > > > > > >>>> a) I think everyone is agreeing that "Only the last 2 versions > > of > > > a > > > > > >>>> connector are supported per supported Flink version, with only > > the > > > > > >> latest > > > > > >>>> version receiving new features". In the current situation, > that > > > > means > > > > > >> that > > > > > >>>> Flink 1.14 and Flink 1.15 would be supported for connectors. > > This > > > > > >> results > > > > > >>>> in a maximum of 4 supported connector versions. > > > > > >>>> > > > > > >>>> b1) In an ideal world, I would have liked Flink's APIs that > are > > > used > > > > > by > > > > > >>>> connectors to be versioned (that's why there's now a Sink V1 > > and a > > > > > Sink > > > > > >>>> V2). However, we're not there yet. > > > > > >>>> > > > > > >>>> b2) With regards to the remark of using @Interal APIs, one > thing > > > > that > > > > > we > > > > > >>>> agreed to in previous discussions is that connectors shouldn't > > > need > > > > to > > > > > >> rely > > > > > >>>> on @Interal APIs so that the connector surface also > stabilizes. > > > > > >>>> > > > > > >>>> b3) In the end, I think what matters the most is the user's > > > > perception > > > > > >> on > > > > > >>>> versioning. So the first thing to establish would be the > > > versioning > > > > > for > > > > > >>>> connectors itself. So you would indeed have a > > <major.minor.patch> > > > > > >> scheme. > > > > > >>>> Next is the compatibility of that scheme with a version of > > Flink. > > > I > > > > do > > > > > >> like > > > > > >>>> Chesnay's approach for using the Scala suffixes idea. So you > > would > > > > > have > > > > > >>>> <major.minor.patch connector>_<major.minor Flink version>. In > > the > > > > > >> currently > > > > > >>>> externalized Elasticsearch connector, we would end up with > > > > 3.0.0_1.14 > > > > > >> and > > > > > >>>> 3.0.0_1.15 as first released versions. If a new Flink version > > > would > > > > be > > > > > >>>> released that doesn't require code changes to the connector, > the > > > > > >> released > > > > > >>>> version would be 3.0.0_1.16. That means that there have been > no > > > > > >> connector > > > > > >>>> code changes (no patches, no new features) when comparing this > > > > across > > > > > >>>> different Flink versions. > > > > > >>>> > > > > > >>>> b4) Now using the example that Chesnay provided (yet slightly > > > > modified > > > > > >> to > > > > > >>>> match it with the Elasticsearch example I've used above), > there > > > > exists > > > > > >> an > > > > > >>>> Elasticsearch connector 3.0.0_1.15. Now in Flink 1.16, > there's a > > > new > > > > > API > > > > > >>>> that we want to use, which is a test util. It would result in > > > > version > > > > > >>>> 3.1.0_1.16 for the new Flink version. Like Chesnay said, for > the > > > > sake > > > > > of > > > > > >>>> argument, at the same time we also had some pending changes > for > > > the > > > > > 1.15 > > > > > >>>> connector (let's say exclusive to 1.15; some workaround for a > > bug > > > or > > > > > >> smth), > > > > > >>>> so we would also end up with 3.1.0-1.15. I agree with Danny > that > > > we > > > > > >> should > > > > > >>>> avoid this situation: the perception of the user would be that > > > > there's > > > > > >> no > > > > > >>>> divergence between the 3.1.0 version, except the compatible > > Flink > > > > > >> version. > > > > > >>>> I really am wondering how often we will run in that situation. > > > From > > > > > what > > > > > >>>> I've seen so far with connectors is that bug fixes always end > up > > > in > > > > > both > > > > > >>>> the release branch and the master branch. The only exceptions > > are > > > > test > > > > > >>>> stabilities or documentation fixes, but if we only resolve > > these, > > > > they > > > > > >>>> wouldn't need to be released. If such a special occasion would > > > > occur, > > > > > I > > > > > >>>> would be inclined to go for a hotfix approach, where you would > > end > > > > up > > > > > >> with > > > > > >>>> 3.0.0.1_1.15. > > > > > >>>> > > > > > >>>> c) Branch wise, I think we should end up with > <major.minor.patch > > > > > >>>> connector>_<major.minor Flink version>. So again the > > Elasticsearch > > > > > >> example, > > > > > >>>> at this moment there would be 3.0.0_1.14 and 3.0.0_1.15 > > branches. > > > > > >>>> > > > > > >>>> Best regards, > > > > > >>>> > > > > > >>>> Martijn > > > > > >>>> > > > > > > > > > > > > > > > > > > > >
