Re: AppVeyor for Windows compatibility testing

Hi, On Jenkins, we can target Windows executor (already available). Probably the easiest way. Regards JB On 10/19/2016 11:54 PM, Lukasz Cwik wrote: I noticed that the Maven exec plugin was using AppVeyor to get testing to occur on windows. Since this is currently a gap in our coverage today, i

Re: Release Guide

Hi, well done. As already discussed, it looks good to me ;) Regards JB On 10/20/2016 01:24 AM, Davor Bonaci wrote: Hi everybody, As a project, I think we should have a Release Guide to document the process, have consistent releases, on-board additional release managers, and generally share kn

Placement of temporary files by FileBasedSink

Hello, This is a continuation of the discussion on PR https://github.com/apache/incubator-beam/pull/1050 which turned out more complex than expected. Short summary: Currently FileBasedSink, when writing to /path/to/foo (in practice, /path/to/foo-x-of-y where y is the total number of o

Re: AppVeyor for Windows compatibility testing

I think we should use Apache Jenkins to get this coverage. It supports both cross-platform and cross-JDK coverage. It should be relatively straightforward to get this enabled. On Wed, Oct 19, 2016 at 2:54 PM, Lukasz Cwik wrote: > I noticed that the Maven exec plugin was using AppVeyor to get tes

Release Guide

Hi everybody, As a project, I think we should have a Release Guide to document the process, have consistent releases, on-board additional release managers, and generally share knowledge. It is also one of the project graduation guidelines. Dan and I wrote a draft version, documenting the process w

AppVeyor for Windows compatibility testing

I noticed that the Maven exec plugin was using AppVeyor to get testing to occur on windows. Since this is currently a gap in our coverage today, is this something we can enable much like our Travis CI for the Apache Beam project?

Jenkins build is back to normal : beam_Release_NightlySnapshot #204

See

Re: [DISCUSS] Sources and Runners

On Wed, Oct 19, 2016 at 11:00 AM, Kenneth Knowles wrote: > I wanted to attempt to explicitly answer Raghu's question by saying that I > think Dan's starting points imply that the recommended behavior for start() > and advance() is to be "non-blocking" in the sense that they return quickly > if in

Re: [DISCUSS] Sources and Runners

I wanted to pull out the sub-thread that isn't about testing, parapharased: Amit: "Dan laid out these points: readers should return ASAP, runners may poll as they see fit [including quickly if they think the reader is in start-up time], runners need to be OK with startup delay" Raghu: "What is the

Re: [DISCUSS] Sources and Runners

Hi FYI when working on IO I already setup a docker image that I'm using for integration test. The IO unit tests embed and bootstrap the IO resources when possible. For instance JmsIO unit tests start a embedded ActiveMQ broker. However I also have a ActiveMQ docker image that I use for integra

Re: [DISCUSS] Sources and Runners

Hadoop FS has the local file system implementation that can be used for testing ("file" URL, no service needed). Thanks On Wed, Oct 19, 2016 at 10:43 AM, Amit Sela wrote: > Oh cool, that didn't exist in 0.8 I think, but anything that is Kafka > native is best. > I'm pretty sure there's an embed

Re: [DISCUSS] Sources and Runners

It will be very useful for existing KafkaIOTest as well. MockConsumer we use is too primitive. ~ 50% of KafkaIOTest deals with MockConsumer. On Wed, Oct 19, 2016 at 10:43 AM, Amit Sela wrote: > Oh cool, that didn't exist in 0.8 I think, but anything that is Kafka > native is best. > I'm pretty s

Re: Simplifying User-Defined Metrics in Beam

On Thu, Oct 13, 2016 at 2:27 AM Aljoscha Krettek wrote: I finally found the time to have a look. :-) The API looks very good! (It's very similar to an API we recently added to Flink, which is inspired by the same Codahale/Dropwizard metrics). About the semantics, the "A", "B" and "C" you m

Re: [DISCUSS] Sources and Runners

Oh cool, that didn't exist in 0.8 I think, but anything that is Kafka native is best. I'm pretty sure there's an embedded HDFS for testing as well. While embedded Kafka/HDFS won't reflect "real-life" distributed environment, it could be a good place to start and provide some basic functional testi

Re: [DISCUSS] Sources and Runners

https://github.com/apache/kafka/blob/trunk/streams/src/test/java/org/apache/kafka/streams/integration/utils/EmbeddedKafkaCluster.java This is currently used in one of our repos and it comes as part of one of kafka libs. On Wed, Oct 19, 2016 at 10:49 PM, Amit Sela wrote: > The SparkRunner actual

Re: [jira] [Commented] (BEAM-755) beam-runners-core-java NeedsRunner tests not executing

At the point in time this was created, there were `NeedsRunner` tests. On Wed, Oct 19, 2016 at 9:34 AM, Kenneth Knowles (JIRA) wrote: > > [ https://issues.apache.org/jira/browse/BEAM-755?page=com. > atlassian.jira.plugin.system.issuetabpanels:comment- > tabpanel&focusedCommentId=15589182#com

Re: [DISCUSS] Sources and Runners

The SparkRunner actually has an embedded Kafka for its unit tests. On Wed, Oct 19, 2016, 20:16 Thomas Weise wrote: > Kafka can be embedded for the integration testing, which should > significantly simplify the setup. > > Here is an example I found: > > https://gist.github.com/fjavieralba/7930018

Re: [DISCUSS] Sources and Runners

Kafka can be embedded for the integration testing, which should significantly simplify the setup. Here is an example I found: https://gist.github.com/fjavieralba/7930018 Thanks, Thomas On Wed, Oct 19, 2016 at 9:44 AM, Dan Halperin wrote: > My thoughts: > > * It's worth reading the Beam tes

Re: [DISCUSS] Sources and Runners

My thoughts: * It's worth reading the Beam testing document that Jason Kuster wrote! * Beam already has support for "End-to-end" integration tests, of examples (e.g., WordCountIT

Re: Exploring Performance Testing

Do you want a benchmark to identify regressions or one to spur competition between the different implementations and let them battle it out for supremacy? You are going to get people using it for both no matter what you do so you should plan on supporting both. The are several problems that need

Re: [DISCUSS] Sources and Runners

+1 those are probably the most used sources. Hadoop FS has a number of different implementations, HDFS is one of them. On Wed, Oct 19, 2016 at 2:55 AM, Amit Sela wrote: > I agree with Aljoscha about Kafka. > > How about having one integration test for BoundedSource and one for > UnboundedSource

Re: [DISCUSS] Sources and Runners

I agree with Aljoscha about Kafka. How about having one integration test for BoundedSource and one for UnboundedSource ? from apache perspective it makes sense to test this end-to-end on HDFS and Kafka (respectively). On Wed, Oct 19, 2016 at 11:34 AM Aljoscha Krettek wrote: > +Jason, looping hi

Re: [DISCUSS] Sources and Runners

+Jason, looping him in directly because he might have an opinion on what I'm going to say. Should we maybe add integration tests that verify that all runners can correctly read from and write to an external system in a complete Pipeline. At least for Kafka, which seems to be the most used option i