We have two different topics in cancelled vote thread. Let's initiate another threads to continue discussion.
- Jungtaek Lim (HeartSaVioR) 2018년 1월 27일 (토) 오후 11:56, Stig Rohde Døssing <stigdoess...@gmail.com>님이 작성: > I don't think storm-kafka-client needs its own project, but I like the idea > of decoupling storm-kafka-client from the release cadence of Storm. > Primarily because it would allow us to release more frequently, but also > because I think most of the time there's no tight coupling between the > Storm minor/patch version and storm-kafka-client. As far as I know > storm-kafka-client 1.2.0 runs fine on other Storm 1.x versions. If > storm-kafka-client released independently, we could probably get away with > maintaining only 1.x and 2.x compatible versions. I also see some value in > being able to do breaking changes independently of Storm's major version, > without breaking semantic versioning. People clearly expect us to not > introduce breaking changes in minor/patch releases (e.g. Alexandre when > storm-kafka-client 1.2.0 wouldn't run with his 1.1.1 topology a while > back), but we've had to do that a few times to avoid postponing fixes until > Storm 2.0.0. Releasing independently would also address Erik's concern that > we're releasing a "new" storm-kafka-client 1.1.2 that has known issues. > > If we want to decouple storm-kafka-client's release cycle from Storm, I > don't think we can keep it in the Storm repo. It would get confusing with > branches and release tags. We might try to decouple it in the same way > Maven have decoupled their plugins from the main Maven repo, where the code > for a given plugin is in a separate repository, but the plugin is still > part of the Maven project and uses the Maven mailing list for discussion > and release announcements. > > My main concern for moving storm-kafka-client to another repository would > be how we build against unreleased Storm versions, e.g. 2.0.0. I'm > wondering if anyone has ideas for this? It looks to me like both the Maven > plugins and Bahir are building against only released versions. > > @Hugo > I think at least a few of your points about storm-kafka-client's implied > beta status would have been easier to handle if storm-kafka-client were not > coupled to the Storm release version. It was weird to have a new > connector's initial release be version 1.0.0, and following Storm's release > cycle has prevented us from releasing fixes as often as a new connector > will likely need. I think we could have also saved ourselves a bit of work > by releasing 0.x versions first, since we then wouldn't have had to worry > about backward compatibility when doing major API changes like > https://issues.apache.org/jira/browse/STORM-2548. > > Regarding whether it is a concern when we make breaking changes in > storm-kafka-client if it is due to Kafka breaking something: Kafka was free > to include breaking changes, because they were still in the pre-1.0.0 > version range which tells the user to expect breaking changes from time to > time. When we release breaking changes in a 1.0.y version, it defeats the > purpose of semantic versioning, which is to tell the user upgrading whether > an upgrade can be done freely, or whether they should expect to have to > recompile and maybe reserve some time to upgrade. > > 2018-01-25 23:44 GMT+01:00 Jungtaek Lim <kabh...@gmail.com>: > > > Hugo, > > > > My idea is basically came from Apache bahir. (http://bahir.apache.org/) > It > > was for Apache Spark, but Flink decided to migrate their connectors to > > bahir so it is also for Apache Flink. > > I admit we may want to keep some connectors in core even we split out > > connectors, since moving to another repo. makes connectors hard to be in > > sync with core version, and even has a chance to being forgotten. Kafka > > connector is first class connector, so maybe storm-kafka-client would not > > take this way even we have similar, but we could incubate it and bring to > > core once it's stabilized (like storm-kafka-client in 1.2.0). > > > > I don't think storm-kafka-client has been technically beta. It might be > > considered as beta implicitly in 1.0.x, but we had 1.1.0 in 10 months > ago. > > storm-kafka-client is introduced over a year being included as Storm > 1.0.0 > > implicitly announcing we are no longer beta. Explicitly marking as beta > is > > very important in practice and that's why InterfaceStability annotation > > came in and widely used for big project. (We have started to apply it for > > streams API in 2.0.0: they're marked as @Unstable.) > > > > Thanks, > > Jungtaek Lim (HeartSaVioR) > > > > 2018년 1월 26일 (금) 오전 4:16, Hugo Da Cruz Louro <hlo...@hortonworks.com>님이 > > 작성: > > > > > I am in favor of not incubating storm-kafka-client but rather keep it > as > > > part of the storm project. We can consider supporting a separate > branch, > > > but before we agree to go that route I would like to hear lessons > learned > > > from community members that have been part of similar transitions in > > other > > > projects. > > > > > > As for not back-porting all the storm-kafka-client changes onto 1.0.x > and > > > 1.1.x branches, I expressed my opinion in the JIRA when the this > > discussion > > > first came up. Basically, I don’t think it is feasible to back-port > > > everything. Furthermore, 1.2.0 will be a minor version for which it is > > > reasonable to expect users of earlier versions to upgrade to without > > major > > > hassle. In any software it is expected that there will be divergence > > across > > > minor versions. New versions become available because they have > > > improvements and and sometimes new, small, features in it. If the user > > > wants to benefit from those improvements, she/he will have to upgrade > to > > > the most recent version. Compatibility should be accounted for across > > > versions, but as Stig mentioned, for the most part we believe it has > > been. > > > > > > Although it is possible to copy the 1.x storm-kafka-client changes to > > > 1.1.x and 1.0.x, that same argument could be made for every other > > connector > > > or storm component, and I am not sure it’s a good principle. I just > think > > > that it’s natural and beneficial to the community and the product that > > > users keep upgrading (especially across backwards compatible versions), > > > rather that keeping on patching things up. > > > > > > A few more opinions on storm-kafka-client > > > - Can we identify the pros and cons of keeping Kafka 0.9 dependencies, > > and > > > unless there are very strong reasons not to do so, simply upgrade to a > > more > > > recent version. > > > - Although storm-kafka-client was not labeled as beta, technically it > was > > > beta, and in practice it was used by users as beta. As far as I know, > > users > > > that used it from the beginning were still exploring it, and I don’t > > think > > > that until a lot of the improvements came in it made it to production. > So > > > Beta is a nice label to have behind a component that can cover for some > > > necessary non backwards compatible changes. However, I don’t think it > > would > > > have made a world of difference in practice for storm-kafka-client. > > > - The Kaka community has at times made non backward compatible changes. > > If > > > Kafka itself makes such changes, is it that concerning if > > > storm-kafka-client has to make or or two such non backward compatible > > > changes? > > > > > > Thanks, > > > Hugo > > > > > > On Jan 25, 2018, at 9:47 AM, P. Taylor Goetz <ptgo...@gmail.com > <mailto: > > > ptgo...@gmail.com>> wrote: > > > > > > > > > > > > On Jan 25, 2018, at 12:32 PM, Bobby Evans <ev...@oath.com.INVALID< > > mailto: > > > ev...@oath.com.INVALID>> wrote: > > > > > > To make this happen though someone is going to need to step up and take > > > lead on this. > > > > > > We will need a plan on what to do (is it becoming a separate repo with > a > > > separate release cycle but still a part of the storm project?) > > > Do we plan to spin it off to be an incubator project on its own? > > > > > > I don’t think it’s necessary to spin it off as a separate incubator > > > project, that seems like overkill and would be weird from a community > > > perspective. > > > > > > The path of least resistance might be a separate git repo, or just > > keeping > > > it in the current Storm repo and decoupling it from the main > > build/release > > > process so it can be independently released. > > > > > > -Taylor > > > > > > > > > > > >
