Re: [PROPOSAL] Introduce beam-sdks-java gradle project

Hi JB, if I understood correctly, you already have a living branch with the required changes. I think it will helpful if you were able to share that to support your proposal. Best, michel On Tue, Apr 2, 2019 at 5:02 PM Jean-Baptiste Onofré wrote: > Hi Michael, > > no problem for the thread,

[Forked] BEAM-4046 (was [PROPOSAL] Introduce beam-sdks-java gradle project)

Hi, agree with Kenn, that this issue at least renders the default implementation difficult to use. Although in the example given, i.e. having sdks/java/core and sdks/py/core, I am unsure, whether it will impose a problem. As far as I understand until now, the issue triggers on dependency declar

Re: kafka 0.9 support

I withdraw my concern -- checked on info on the cluster I will eventually access. It is on 0.8, so I was speaking too soon. Can't speak to rest of user base. On Tue, Apr 2, 2019 at 11:03 AM Raghu Angadi wrote: > Thanks to David Morávek for pointing out possible improvement to KafkaIO > for dro

Re: Quieten javadoc generation

+1 to suppress warnings globally. If we care about an issue, it should be error. On Tue, Apr 2, 2019 at 5:38 AM Alexey Romanenko wrote: > +1 to suppress such warnings globally. IMO, usually, meaningful Javadoc > description is quite enough to understand what this method does. > > On 1 Apr 2019,

Re: kafka 0.9 support

Thanks to David Morávek for pointing out possible improvement to KafkaIO for dropping support for 0.9 since it avoids having a second consumer just to fetch latest offsets for backlog. Ideally we should be dropping 0.9 support for next major release, in fact better to drop versions before 0.10.1 a

Re: Debugging :beam-sdks-java-io-hadoop-input-format:test

Hi everyone, I created BEAM-6974 . This test and beam-sdks-java-io-cassandra tests fail often in our Pre-Commit jobs. Can someone look into this? Thank you, Mikhail. On Thu, Mar 28, 2019 at 12:31 PM Mikhail Gryzykhin wrote: > I've seen it coupl

Re: kafka 0.9 support

We're still using Kafka 0.10 a lot, similar as 0.9 IMO. To expand multiple versions in KafkaIO is quite complex now, and it confuses users which is supported / which is not. I would prefer to support Kafka 2.0+ only in the latest version. For old versions, there're some options: 1). document Kafka-

Re: [DISCUSS] SplittableDoFn Java SDK User Facing API

I was able to update the failing Watch transform in https://github.com/apache/beam/pull/8146 and this has now been merged. On Mon, Mar 18, 2019 at 10:32 AM Lukasz Cwik wrote: > Thanks Kenn, based upon the error message there was a small amount of code > that I missed when updating the code. I'll

Re: [PROPOSAL] Introduce beam-sdks-java gradle project

I didn't know that https://github.com/gradle/gradle/issues/847 existed but the description of the issues people are having are similar to what was discovered during the gradle migration. On Tue, Apr 2, 2019 at 8:02 AM Jean-Baptiste Onofré wrote: > Hi Michael, > > no problem for the thread, that'

Re: kafka 0.9 support

KafkaIO is marked as Experimental, and the comment already warns that 0.9 support might be removed. I think that if users still rely on Kafka 0.9 we should leave a fork (renamed) of the IO in the tree for 0.9, but we can definitely remove 0.9 support from the main IO if we want, especially if it's

Re: kafka 0.9 support

> How are multiple versions of Kafka supported? Are they all in one client, or > is there a case for forks like ElasticSearchIO? They are supported in one client but we have additional “ConsumerSpEL” adapter which unifies interface difference among different Kafka client versions (mostly to sup

Re: Beam contribution

+Ahmed I have added you as a contributor. It seems as though Ahmed had just picked up BEAM-3489 yesterday. Reach out to Ahmed if you would like to help them out with the task. Was TimerReceiverTest failing reliably when performing a parallel build or is it flaky? I have asked Chamikara to take

Re: kafka 0.9 support

If users are still stuck on Kafka 0.9, that's an argument that we should continue supporting it. however, if it's making it hard for us to support KafkaIO, maybe we should consider splitting it? We could fork off KafkaIO0.9, and remove 0.9 support from the regular KafkaIO. On Mon, Apr 1, 2019 at 5

Beam contribution

Hi All! I am Csabi, I would be happy to contribute to Beam. Could you grant me contributor role and assign issue BEAM-3489 to me? My user name is " csabakassai". After I checked out the code and tried to do a gradle check I found these issues:

Re: [PROPOSAL] Introduce beam-sdks-java gradle project

Hi Michael, no problem for the thread, that's the goal of the mailing list ;) And yes, you got my idea about a "meta" module: easy way of building the "whole" Java SDK. The purpose is not to create a uber jar, but more to simplify the build for Java SDK developers. Do you want me to complete yo

Re: [PROPOSAL] Introduce beam-sdks-java gradle project

Going to fork the BEAM-4046 discussion. And, JB, I apologise for hijacking your thread. As for the original question, I understood a request for a meta project which will enable easier handling of java projects. E.g. instead of requiring the user to call ./gradlew module1:build module2:build

Re: Deprecating Avro for fastavro on Python 3

I agree with Ahmet. Fastavro seems to be well maintained and has good, tested compatibility. Unless we expect significant performance improvements in the standard Avro Python package (a significant undertaking, likely not one we have the bandwidth to take on, and my impression is that it's histori

Re: Quieten javadoc generation

+1 to suppress such warnings globally. IMO, usually, meaningful Javadoc description is quite enough to understand what this method does. > On 1 Apr 2019, at 18:21, Kenneth Knowles wrote: > > Personally, I would like to suppress the warnings globally. I think requiring > javadoc everywhere is a

Re: Deprecating Avro for fastavro on Python 3

Hi all, Thank you for the feedback. Looking at the responses, it seems like there is a consensus to move forward with fastavro as the default implementation on Python 3. There are 2 questions left however: - Should fastavro also become the default implementation on Python 2? This is a trade-off b