-1 (non-binding) Please include https://issues.apache.org/jira/browse/STORM-2918
On Jan 29, 2018 12:03 AM, "Jungtaek Lim" <[email protected]> wrote: > We have two different topics in cancelled vote thread. Let's initiate > another threads to continue discussion. > > - Jungtaek Lim (HeartSaVioR) > > 2018년 1월 27일 (토) 오후 11:56, Stig Rohde Døssing <[email protected]>님이 > 작성: > > > I don't think storm-kafka-client needs its own project, but I like the > idea > > of decoupling storm-kafka-client from the release cadence of Storm. > > Primarily because it would allow us to release more frequently, but also > > because I think most of the time there's no tight coupling between the > > Storm minor/patch version and storm-kafka-client. As far as I know > > storm-kafka-client 1.2.0 runs fine on other Storm 1.x versions. If > > storm-kafka-client released independently, we could probably get away > with > > maintaining only 1.x and 2.x compatible versions. I also see some value > in > > being able to do breaking changes independently of Storm's major version, > > without breaking semantic versioning. People clearly expect us to not > > introduce breaking changes in minor/patch releases (e.g. Alexandre when > > storm-kafka-client 1.2.0 wouldn't run with his 1.1.1 topology a while > > back), but we've had to do that a few times to avoid postponing fixes > until > > Storm 2.0.0. Releasing independently would also address Erik's concern > that > > we're releasing a "new" storm-kafka-client 1.1.2 that has known issues. > > > > If we want to decouple storm-kafka-client's release cycle from Storm, I > > don't think we can keep it in the Storm repo. It would get confusing with > > branches and release tags. We might try to decouple it in the same way > > Maven have decoupled their plugins from the main Maven repo, where the > code > > for a given plugin is in a separate repository, but the plugin is still > > part of the Maven project and uses the Maven mailing list for discussion > > and release announcements. > > > > My main concern for moving storm-kafka-client to another repository would > > be how we build against unreleased Storm versions, e.g. 2.0.0. I'm > > wondering if anyone has ideas for this? It looks to me like both the > Maven > > plugins and Bahir are building against only released versions. > > > > @Hugo > > I think at least a few of your points about storm-kafka-client's implied > > beta status would have been easier to handle if storm-kafka-client were > not > > coupled to the Storm release version. It was weird to have a new > > connector's initial release be version 1.0.0, and following Storm's > release > > cycle has prevented us from releasing fixes as often as a new connector > > will likely need. I think we could have also saved ourselves a bit of > work > > by releasing 0.x versions first, since we then wouldn't have had to worry > > about backward compatibility when doing major API changes like > > https://issues.apache.org/jira/browse/STORM-2548. > > > > Regarding whether it is a concern when we make breaking changes in > > storm-kafka-client if it is due to Kafka breaking something: Kafka was > free > > to include breaking changes, because they were still in the pre-1.0.0 > > version range which tells the user to expect breaking changes from time > to > > time. When we release breaking changes in a 1.0.y version, it defeats the > > purpose of semantic versioning, which is to tell the user upgrading > whether > > an upgrade can be done freely, or whether they should expect to have to > > recompile and maybe reserve some time to upgrade. > > > > 2018-01-25 23:44 GMT+01:00 Jungtaek Lim <[email protected]>: > > > > > Hugo, > > > > > > My idea is basically came from Apache bahir. (http://bahir.apache.org/ > ) > > It > > > was for Apache Spark, but Flink decided to migrate their connectors to > > > bahir so it is also for Apache Flink. > > > I admit we may want to keep some connectors in core even we split out > > > connectors, since moving to another repo. makes connectors hard to be > in > > > sync with core version, and even has a chance to being forgotten. Kafka > > > connector is first class connector, so maybe storm-kafka-client would > not > > > take this way even we have similar, but we could incubate it and bring > to > > > core once it's stabilized (like storm-kafka-client in 1.2.0). > > > > > > I don't think storm-kafka-client has been technically beta. It might be > > > considered as beta implicitly in 1.0.x, but we had 1.1.0 in 10 months > > ago. > > > storm-kafka-client is introduced over a year being included as Storm > > 1.0.0 > > > implicitly announcing we are no longer beta. Explicitly marking as beta > > is > > > very important in practice and that's why InterfaceStability annotation > > > came in and widely used for big project. (We have started to apply it > for > > > streams API in 2.0.0: they're marked as @Unstable.) > > > > > > Thanks, > > > Jungtaek Lim (HeartSaVioR) > > > > > > 2018년 1월 26일 (금) 오전 4:16, Hugo Da Cruz Louro <[email protected] > >님이 > > > 작성: > > > > > > > I am in favor of not incubating storm-kafka-client but rather keep it > > as > > > > part of the storm project. We can consider supporting a separate > > branch, > > > > but before we agree to go that route I would like to hear lessons > > learned > > > > from community members that have been part of similar transitions in > > > other > > > > projects. > > > > > > > > As for not back-porting all the storm-kafka-client changes onto 1.0.x > > and > > > > 1.1.x branches, I expressed my opinion in the JIRA when the this > > > discussion > > > > first came up. Basically, I don’t think it is feasible to back-port > > > > everything. Furthermore, 1.2.0 will be a minor version for which it > is > > > > reasonable to expect users of earlier versions to upgrade to without > > > major > > > > hassle. In any software it is expected that there will be divergence > > > across > > > > minor versions. New versions become available because they have > > > > improvements and and sometimes new, small, features in it. If the > user > > > > wants to benefit from those improvements, she/he will have to upgrade > > to > > > > the most recent version. Compatibility should be accounted for across > > > > versions, but as Stig mentioned, for the most part we believe it has > > > been. > > > > > > > > Although it is possible to copy the 1.x storm-kafka-client changes to > > > > 1.1.x and 1.0.x, that same argument could be made for every other > > > connector > > > > or storm component, and I am not sure it’s a good principle. I just > > think > > > > that it’s natural and beneficial to the community and the product > that > > > > users keep upgrading (especially across backwards compatible > versions), > > > > rather that keeping on patching things up. > > > > > > > > A few more opinions on storm-kafka-client > > > > - Can we identify the pros and cons of keeping Kafka 0.9 > dependencies, > > > and > > > > unless there are very strong reasons not to do so, simply upgrade to > a > > > more > > > > recent version. > > > > - Although storm-kafka-client was not labeled as beta, technically it > > was > > > > beta, and in practice it was used by users as beta. As far as I know, > > > users > > > > that used it from the beginning were still exploring it, and I don’t > > > think > > > > that until a lot of the improvements came in it made it to > production. > > So > > > > Beta is a nice label to have behind a component that can cover for > some > > > > necessary non backwards compatible changes. However, I don’t think it > > > would > > > > have made a world of difference in practice for storm-kafka-client. > > > > - The Kaka community has at times made non backward compatible > changes. > > > If > > > > Kafka itself makes such changes, is it that concerning if > > > > storm-kafka-client has to make or or two such non backward compatible > > > > changes? > > > > > > > > Thanks, > > > > Hugo > > > > > > > > On Jan 25, 2018, at 9:47 AM, P. Taylor Goetz <[email protected] > > <mailto: > > > > [email protected]>> wrote: > > > > > > > > > > > > > > > > On Jan 25, 2018, at 12:32 PM, Bobby Evans <[email protected]< > > > mailto: > > > > [email protected]>> wrote: > > > > > > > > To make this happen though someone is going to need to step up and > take > > > > lead on this. > > > > > > > > We will need a plan on what to do (is it becoming a separate repo > with > > a > > > > separate release cycle but still a part of the storm project?) > > > > Do we plan to spin it off to be an incubator project on its own? > > > > > > > > I don’t think it’s necessary to spin it off as a separate incubator > > > > project, that seems like overkill and would be weird from a community > > > > perspective. > > > > > > > > The path of least resistance might be a separate git repo, or just > > > keeping > > > > it in the current Storm repo and decoupling it from the main > > > build/release > > > > process so it can be independently released. > > > > > > > > -Taylor > > > > > > > > > > > > > > > > > >
