Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

Hello, We modified the tool based on the new policies discussed in this thread (the latest report ). The major changes include: - Title (summary) of the

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

Thanks all for comments and suggestions. We want to close this thread and start implementing the new policy based on the discussion: 1. Stop assigning JIRAs to the first person listed in the dependency owners files . Instead, cc people on the o

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

+1 on the jira "fix version". The release frequency of dependencies are various, so that using new information such as versions from the Jira closing date to reopen the issues might not be very proper. We could check the fix versions first, and if specified, then reopen the issue in that version's

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

On Wed, Sep 5, 2018 at 12:50 PM Tim Robertson wrote: > Thank you Cham, and everyone for contributing > > Sorry for slow reply to a thread I started, but I've been swamped on non > Beam projects. > > KafkaIO's policy of 'let the user decide exact version at runtime' has >> been quite useful so far

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

Thank you Cham, and everyone for contributing Sorry for slow reply to a thread I started, but I've been swamped on non Beam projects. KafkaIO's policy of 'let the user decide exact version at runtime' has been > quite useful so far. How feasible is that for other connectors? I presume shimming

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

Thanks Cham for putting this together. Also, after modifying the dependency tool based on the policy above, we will close all existing JIRA issues that prevent creating duplicate bugs and stop pushing assignees to upgrade dependencies with old bugs. Please let us know if you have any comments on t

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

Based on this email thread and offline feedback from several folks, current concerns regarding dependency upgrade policy and tooling seems to be following. (1) We have to be careful when upgrading dependencies. For example, we should not create JIRAs for upgrading to dependency versions that have

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

On Tue, Aug 28, 2018 at 12:05 PM Thomas Weise wrote: > I think there is an invalid assumption being made in this discussion, > which is that most projects comply with semantic versioning. The reality in > the open source big data space is unfortunately quite different. Ismaël has > well character

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

I think there is an invalid assumption being made in this discussion, which is that most projects comply with semantic versioning. The reality in the open source big data space is unfortunately quite different. Ismaël has well characterized the situation and HBase isn't an exception. Another indica

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

Thanks for the IO versioning summary. KafkaIO's policy of 'let the user decide exact version at runtime' has been quite useful so far. How feasible is that for other connectors? Also, KafkaIO does not limit itself to minimum features available across all the supported versions. Some of the feature

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

Constrains to existing dependencies is a valid concern and we do not have a good solution for this currently. One way to handle this is be simply to close automatically created JIRAs with a comment and the tool will not try to create further JIRAs for the same dependency after this. But we should b

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

The Beam SQL module faces similar problems, several of our dependencies are constrained by maintaining compatibility with versions used by Calcite. We've written tests to detect some of these incompatibilities. Could we add integration tests for these major hadoop distros that ensure we maintain co

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

Thanks Tim for raising this and Thanks JB and Ismaël for all the great points. I agree that one size fit all solution will not work when it comes to dependencies. Based on past examples, clearly there are many cases where we should proceed with caution and upgrade dependencies with care. That sai

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

I think we should refine the strategy on dependencies discussed recently. Sorry to come late with this (I did not follow closely the previous discussion), but the current approach is clearly not in line with the industry reality (at least not for IO connectors + Hadoop + Spark/Flink use). A really

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

Hi Tim, regarding the IO, while ago (at the incubator time of the project), we discussed how to deal with different versions of the backend API and dependencies. I proposed to have a release cycle per IO and have a subproject per IO version, like for instance: sdks/java/io/elasticsearch-5 sdks/ja

[DISCUSS] Versioning, Hadoop related dependencies and enterprise users

Hi folks, I'd like to revisit the discussion around our versioning policy specifically for the Hadoop ecosystem and make sure we are aware of the implications. As an example our policy today would have us on HBase 2.1 and I have reminders to address this. However, currently the versions of HBase