-1 to include STORM-2918 <https://issues.apache.org/jira/browse/STORM-2918>
Thanks, Satish. On Thu, Feb 1, 2018 at 2:28 PM, Artem Ervits <[email protected]> wrote: > -1 (non-binding) > Please include https://issues.apache.org/jira/browse/STORM-2918 > > On Jan 29, 2018 12:03 AM, "Jungtaek Lim" <[email protected]> wrote: > > > We have two different topics in cancelled vote thread. Let's initiate > > another threads to continue discussion. > > > > - Jungtaek Lim (HeartSaVioR) > > > > 2018년 1월 27일 (토) 오후 11:56, Stig Rohde Døssing <[email protected]>님이 > > 작성: > > > > > I don't think storm-kafka-client needs its own project, but I like the > > idea > > > of decoupling storm-kafka-client from the release cadence of Storm. > > > Primarily because it would allow us to release more frequently, but > also > > > because I think most of the time there's no tight coupling between the > > > Storm minor/patch version and storm-kafka-client. As far as I know > > > storm-kafka-client 1.2.0 runs fine on other Storm 1.x versions. If > > > storm-kafka-client released independently, we could probably get away > > with > > > maintaining only 1.x and 2.x compatible versions. I also see some value > > in > > > being able to do breaking changes independently of Storm's major > version, > > > without breaking semantic versioning. People clearly expect us to not > > > introduce breaking changes in minor/patch releases (e.g. Alexandre when > > > storm-kafka-client 1.2.0 wouldn't run with his 1.1.1 topology a while > > > back), but we've had to do that a few times to avoid postponing fixes > > until > > > Storm 2.0.0. Releasing independently would also address Erik's concern > > that > > > we're releasing a "new" storm-kafka-client 1.1.2 that has known issues. > > > > > > If we want to decouple storm-kafka-client's release cycle from Storm, I > > > don't think we can keep it in the Storm repo. It would get confusing > with > > > branches and release tags. We might try to decouple it in the same way > > > Maven have decoupled their plugins from the main Maven repo, where the > > code > > > for a given plugin is in a separate repository, but the plugin is still > > > part of the Maven project and uses the Maven mailing list for > discussion > > > and release announcements. > > > > > > My main concern for moving storm-kafka-client to another repository > would > > > be how we build against unreleased Storm versions, e.g. 2.0.0. I'm > > > wondering if anyone has ideas for this? It looks to me like both the > > Maven > > > plugins and Bahir are building against only released versions. > > > > > > @Hugo > > > I think at least a few of your points about storm-kafka-client's > implied > > > beta status would have been easier to handle if storm-kafka-client were > > not > > > coupled to the Storm release version. It was weird to have a new > > > connector's initial release be version 1.0.0, and following Storm's > > release > > > cycle has prevented us from releasing fixes as often as a new connector > > > will likely need. I think we could have also saved ourselves a bit of > > work > > > by releasing 0.x versions first, since we then wouldn't have had to > worry > > > about backward compatibility when doing major API changes like > > > https://issues.apache.org/jira/browse/STORM-2548. > > > > > > Regarding whether it is a concern when we make breaking changes in > > > storm-kafka-client if it is due to Kafka breaking something: Kafka was > > free > > > to include breaking changes, because they were still in the pre-1.0.0 > > > version range which tells the user to expect breaking changes from time > > to > > > time. When we release breaking changes in a 1.0.y version, it defeats > the > > > purpose of semantic versioning, which is to tell the user upgrading > > whether > > > an upgrade can be done freely, or whether they should expect to have to > > > recompile and maybe reserve some time to upgrade. > > > > > > 2018-01-25 23:44 GMT+01:00 Jungtaek Lim <[email protected]>: > > > > > > > Hugo, > > > > > > > > My idea is basically came from Apache bahir. ( > http://bahir.apache.org/ > > ) > > > It > > > > was for Apache Spark, but Flink decided to migrate their connectors > to > > > > bahir so it is also for Apache Flink. > > > > I admit we may want to keep some connectors in core even we split out > > > > connectors, since moving to another repo. makes connectors hard to be > > in > > > > sync with core version, and even has a chance to being forgotten. > Kafka > > > > connector is first class connector, so maybe storm-kafka-client would > > not > > > > take this way even we have similar, but we could incubate it and > bring > > to > > > > core once it's stabilized (like storm-kafka-client in 1.2.0). > > > > > > > > I don't think storm-kafka-client has been technically beta. It might > be > > > > considered as beta implicitly in 1.0.x, but we had 1.1.0 in 10 months > > > ago. > > > > storm-kafka-client is introduced over a year being included as Storm > > > 1.0.0 > > > > implicitly announcing we are no longer beta. Explicitly marking as > beta > > > is > > > > very important in practice and that's why InterfaceStability > annotation > > > > came in and widely used for big project. (We have started to apply it > > for > > > > streams API in 2.0.0: they're marked as @Unstable.) > > > > > > > > Thanks, > > > > Jungtaek Lim (HeartSaVioR) > > > > > > > > 2018년 1월 26일 (금) 오전 4:16, Hugo Da Cruz Louro <[email protected] > > >님이 > > > > 작성: > > > > > > > > > I am in favor of not incubating storm-kafka-client but rather keep > it > > > as > > > > > part of the storm project. We can consider supporting a separate > > > branch, > > > > > but before we agree to go that route I would like to hear lessons > > > learned > > > > > from community members that have been part of similar transitions > in > > > > other > > > > > projects. > > > > > > > > > > As for not back-porting all the storm-kafka-client changes onto > 1.0.x > > > and > > > > > 1.1.x branches, I expressed my opinion in the JIRA when the this > > > > discussion > > > > > first came up. Basically, I don’t think it is feasible to back-port > > > > > everything. Furthermore, 1.2.0 will be a minor version for which it > > is > > > > > reasonable to expect users of earlier versions to upgrade to > without > > > > major > > > > > hassle. In any software it is expected that there will be > divergence > > > > across > > > > > minor versions. New versions become available because they have > > > > > improvements and and sometimes new, small, features in it. If the > > user > > > > > wants to benefit from those improvements, she/he will have to > upgrade > > > to > > > > > the most recent version. Compatibility should be accounted for > across > > > > > versions, but as Stig mentioned, for the most part we believe it > has > > > > been. > > > > > > > > > > Although it is possible to copy the 1.x storm-kafka-client changes > to > > > > > 1.1.x and 1.0.x, that same argument could be made for every other > > > > connector > > > > > or storm component, and I am not sure it’s a good principle. I just > > > think > > > > > that it’s natural and beneficial to the community and the product > > that > > > > > users keep upgrading (especially across backwards compatible > > versions), > > > > > rather that keeping on patching things up. > > > > > > > > > > A few more opinions on storm-kafka-client > > > > > - Can we identify the pros and cons of keeping Kafka 0.9 > > dependencies, > > > > and > > > > > unless there are very strong reasons not to do so, simply upgrade > to > > a > > > > more > > > > > recent version. > > > > > - Although storm-kafka-client was not labeled as beta, technically > it > > > was > > > > > beta, and in practice it was used by users as beta. As far as I > know, > > > > users > > > > > that used it from the beginning were still exploring it, and I > don’t > > > > think > > > > > that until a lot of the improvements came in it made it to > > production. > > > So > > > > > Beta is a nice label to have behind a component that can cover for > > some > > > > > necessary non backwards compatible changes. However, I don’t think > it > > > > would > > > > > have made a world of difference in practice for storm-kafka-client. > > > > > - The Kaka community has at times made non backward compatible > > changes. > > > > If > > > > > Kafka itself makes such changes, is it that concerning if > > > > > storm-kafka-client has to make or or two such non backward > compatible > > > > > changes? > > > > > > > > > > Thanks, > > > > > Hugo > > > > > > > > > > On Jan 25, 2018, at 9:47 AM, P. Taylor Goetz <[email protected] > > > <mailto: > > > > > [email protected]>> wrote: > > > > > > > > > > > > > > > > > > > > On Jan 25, 2018, at 12:32 PM, Bobby Evans <[email protected]< > > > > mailto: > > > > > [email protected]>> wrote: > > > > > > > > > > To make this happen though someone is going to need to step up and > > take > > > > > lead on this. > > > > > > > > > > We will need a plan on what to do (is it becoming a separate repo > > with > > > a > > > > > separate release cycle but still a part of the storm project?) > > > > > Do we plan to spin it off to be an incubator project on its own? > > > > > > > > > > I don’t think it’s necessary to spin it off as a separate incubator > > > > > project, that seems like overkill and would be weird from a > community > > > > > perspective. > > > > > > > > > > The path of least resistance might be a separate git repo, or just > > > > keeping > > > > > it in the current Storm repo and decoupling it from the main > > > > build/release > > > > > process so it can be independently released. > > > > > > > > > > -Taylor > > > > > > > > > > > > > > > > > > > > > > > > >
