Hi,
In general connectors are independent of Storm run-time for most
parts. I.e if the APIs are not changed (storm-core or trident haven't changed
in years except the package re-name). You can take the latest connector and run
in storm 1.0 or higher. So the users doesn't need to upgrade their storm
cluster just to get a latest connector upgrade. Which they might be doing it
but by making the release separate and stating the minimum supported storm
version for the connectors will help the users.
This makes it easier for the connectors to be released independently of the
core/run-time and makes it easy for them to be fixed and released more often.
But moving them to Bahir or other external project will make it detached from
Storm itself that it might not see any co-ordination as reviewers from storm
will need to be aware of an external project.
My proposal would be
1. Can we create a sub-project in git under Storm so we can move the connectors
there and everything else related Storm applies there.
2. Can we keep maintaining storm connectors within same repo but different
release module for it .
This is a separate topic but can improve the release timelines if we have
multiple release managers that are handling the maint release and also main
release versions. Its good to have rotation of release managers from PMC so
that everyone will understand the process and can spread the responsibilities.
There are threads started before but don't think they are addressed or any
action item is taken. We should start another thread to discuss this process as
well.
Thanks,
Harsha
On Tue, Jan 30, 2018, at 9:49 AM, Hugo Da Cruz Louro wrote:
> I think that the bahir approach makes sense for connectors that don’t
> fall into the "first class support” category. I am in favor of moving
> such lower adoption connectors and have the interested communities
> support them with the most suitable release cycle. Connectors that are
> idle, such as some examples that Jungtaek gave, we should consider
> removing them altogether, especially if they are so outdated that they
> may not even work.
>
> Mainstream connectors such as storm-kafka-client should be kept in the
> Storm repo. For example, Flink keeps flink-connector-kafka-0.x in the
> Flink repo.
>
> I am in agreement with Jungtaek when he says: "fixing critical bugs in
> storm-kafka-client should trigger release, instead of waiting for Storm
> core to have some fixes to be worth to release”. Storm’s release cadence
> is currently not very high and one can argue that Storm entirely could
> benefit from more frequent releases. If it is sto rm-kafka-client
> triggering those releases, so be it. Moving forward I do not expect the
> storm-kafka-client connector to be subject to so many changes that it
> would warrant its own release cycle.
>
> I also would like to highlight that although storm-kafka-client has been
> the center of this discussion, as it was mentioned in this
> thread<https://goo.gl/VY7QTG>, storm-kafka-client has had a much less
> rocky road to stability compared to for example storm-kafka. Therefore
> it’s worth evaluating if the challenges that we have faced with storm-
> kafka-client have been out of norm for such an important and complex
> feature, and if they warrant significant changes in how we do things.
>
> Thanks,
> Hugo
>
> On Jan 29, 2018, at 9:18 PM, Jungtaek Lim
> <kabh...@gmail.com<mailto:kabh...@gmail.com>> wrote:
>
> Let me add a proof of my opinion: major patch of storm-eventhubs hasn't
> been getting even a comment over 4 months.
> https://github.com/apache/storm/pull/2322
>
> I'd rather want to discuss regarding discontinue supporting officially if
> we no longer interest of, or we don't have resource to support, or any
> valid reasons. If we agree on discontinue supporting officially, we can
> move out to other repo. and let it self maintained. It may be able to get
> attention and have enough contributors so that we feel better to get to
> Storm core Repository again, or it can be silently forgotten. It shouldn't
> affect Storm core repository at any case.
>
> 2018년 1월 30일 (화) 오후 2:03, Jungtaek Lim <kabh...@gmail.com>님이 작성:
>
> If we worry about breaking somethings along with our
> users/consumers/distributors, picking one of less used/updated connector as
> experiment makes more sense to me. It's OK if we want to pick one of most
> active and widely used connector intentionally to accelerate experiment.
>
> Decoupling connectors and moving to other repo. like Bahir will make it
> clear who are having interest of which connectors. storm-eventhubs for
> example, major code contributions were done from MS developers. Now they
> are gone, and I don't know even storm-eventhubs are compatible with recent
> Azure Eventhub. That's just a one of them. I've seen many connectors in
> same, or similar, or possible (say truck number 1) situation.
>
> -Jungtaek Lim (HeartSaVioR)
>
> 2018년 1월 30일 (화) 오후 1:30, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>
>
> On Jan 29, 2018, at 8:03 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
>
>
>
> - Do we ensure they're all maintained?
> -- Did we exclude inactive committers/PMCs for connector's committer
>
> sponsors, and do they have enough committer sponsors after that?
>
>
> Good point. We’ve had some sponsors go silent recently. Maybe ping
> sponsors and ask if they wish to maintain sponsorship?
>
> As a sponsor for a number of connectors, I’ll check on the ones I’ve
> sponsored.
>
> - Do they all worth to keep maintaining in Storm main repository?
>
>
> Again, that’s a question of whether there is user/dev interest.
>
>
> -- Should we trigger release if we find and resolve critical/blocker issue
> from them? If not, why we allow to leave the thing which is in main
> repository as inconsistent state?
>
>
> Some are tied to fairly well established protocols, some target really
> volatile APIs. Bug reports and mailing list activity may not be a good
> status indicator.
>
> Storm’s Kafka integration was the initial model for the “batteries
> included” impetus behind `external`. If we want to evolve how that works,
> why not start there, see what works/doesn’t work, and adapt.
>
> I don’t want to shock our users/consumers/distributors.
>
>
> -Taylor
>
>
>
>
>
>
>
