Re: Capability Matrix

Great job ! By the way, when you use ~ in the matrix, does it mean that it works only in some cases (depending of the pipeline or transform) or it doesn't work as expected ? Just curious for the Aggregators and the meaning in the Beam Model. Thanks, Regards JB On 03/18/2016 12:45 AM, Tyler

Re: Unable to serialize exception running KafkaWindowedWordCountExample

@Dan: You're right that the PipelineOptions shouldn't be cached like this. In this particular wrapper, it was not even necessary. @Jiankang: I've pushed a fix to the repository with a few improvements. Could you please try again? You will have to recompile. Thanks, Max On Thu, Mar 17, 2016 at 8:

Re: Unable to serialize exception running KafkaWindowedWordCountExample

Hi Jiankang, Thanks for reporting again. I'm sorry that you ran into another problem. This example had been working but it has some small problems with the new code base we just migrated to. I've fixed and tested the example and would invite you to try again. Thanks, Max On Thu, Mar 17, 2016 at

Re: [PROPOSAL] MultiLineIO

Have you considered expanding TextIO to support an arbitrary delimiter instead of defining MultiLineIO? https://github.com/apache/incubator-beam/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/TextIO.java#L737 TextIO currently splits on '\n', '\r\n', or '\r'. It seems as though havi

Re: Capability Matrix

Well done. The matrix provides a good basis for improving the existing runners. Moreover, new backends can use it to evaluate capabilities for creating a runner. On Fri, Mar 18, 2016 at 1:15 AM, Jean-Baptiste Onofré wrote: > Catcha, thanks ! > > Regards > JB > > > On 03/18/2016 12:51 AM, Frances

Polishing/renaming & version

Hi team, Maybe you saw my PR about the first step in the process of polishing/renaming Beam to be "Apache compliant". On this PR, we had some discussion about the version that I would like to bring on the mailing list. I did for now is to use 1.5.0-incubating-SNAPSHOT version. We had some p

Re: Capability Matrix

Catcha, thanks ! Regards JB On 03/18/2016 12:51 AM, Frances Perry wrote: That's "partially". Check out the full matrix for complete details: http://beam.incubator.apache.org/capability-matrix/ On Thu, Mar 17, 2016 at 4:50 PM, Jean-Baptiste Onofré wrote: Great job ! By the way, when you use

Re: Unable to serialize exception running KafkaWindowedWordCountExample

+Max for the Flink Runner, and +Luke who wrote most of the initial code around PipelineOptions. The UnboundedFlinkSource is caching the `PipelineOptions` object, here: https://github.com/apache/incubator-beam/blob/071e4dd67021346b0cab2aafa0900ec7e34c4ef8/runners/flink/runner/src/main/java/org/apac

Re: Jenkins build became unstable: beam_MavenVerify #14

Adding Jason explicitly. This is somewhat expected. Thanks Andreas. On Thu, Mar 17, 2016 at 8:08 AM, Andreas Veithen wrote: > It looks like nowadays, when enabling "Use private Maven repository", you > also need to select a strategy. If you want to shield the build against > pollution of the lo

Re: Jenkins build became unstable: beam_MavenVerify #14

Looks like the cached MapR Hadoop dependencies issue is back.. I didn't push anything lately, and the only commit was unrelated. On Thu, Mar 17, 2016 at 4:54 PM Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See > >

Re: Draft Contribution Guide

Hi Frances, thanks, it looks good to me (just proposed a minor suggestion). I see that James should update the website section. @James, let me know if I can help there. Regards JB On 03/17/2016 10:19 PM, Frances Perry wrote: Hi Beamers! We've started a draft

Re: [jira] [Commented] (BEAM-129) Support pubsub IO

Hi Mark, Thanks a lot for reporting this and for working on it! We just migrated all the known issues of the Flink-runner from our previous repository to the BEAM one. Kostas > On Mar 18, 2016, at 6:29 PM, Mark Shields (JIRA) wrote: > > >[ > https://issues.apache.org/jira/browse/BEAM-1

Re: Draft Contribution Guide

Hi Frances, Very nice comprehensive guide. I'll leave some comments in the doc. Cheers, Max On Fri, Mar 18, 2016 at 11:51 AM, Sandeep Deshmukh wrote: > The document captures the process very well and has right amount of details > for newbies too. > > Great work!!! > > Regards, > Sandeep > > On

Re: Capability Matrix

That's "partially". Check out the full matrix for complete details: http://beam.incubator.apache.org/capability-matrix/ On Thu, Mar 17, 2016 at 4:50 PM, Jean-Baptiste Onofré wrote: > Great job ! > > By the way, when you use ~ in the matrix, does it mean that it works only > in some cases (depend

Re: Revision to Pipeline Runner API: 1-Pager

+1 Regards JB On 03/18/2016 03:56 PM, Kenneth Knowles wrote: Hi all, Since I haven't heard any major objections or suggestions in a little while, I'm going to go ahead and put together an actual design doc to share that fleshes out the proposed work a bit more. Kenn On Tue, Mar 15, 2016 at 8

Unable to serialize exception running KafkaWindowedWordCountExample

Hi guys, Failed to run KafkaWindowedWordCountExample with Unable to serialize exception, the stack exception as below: 16/03/17 13:49:09 INFO flink.FlinkPipelineRunner: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 160 files. Enable logging at

Re: Draft Contribution Guide

The document captures the process very well and has right amount of details for newbies too. Great work!!! Regards, Sandeep On Fri, Mar 18, 2016 at 10:46 AM, Siva Kalagarla wrote: > Thanks Frances, This document is helpful for newbies like myself. Will > follow these steps over this weekend.

Re: Revision to Pipeline Runner API: 1-Pager

Hi all, Since I haven't heard any major objections or suggestions in a little while, I'm going to go ahead and put together an actual design doc to share that fleshes out the proposed work a bit more. Kenn On Tue, Mar 15, 2016 at 8:28 AM, Kenneth Knowles wrote: > Hi everyone, > > I have put to

Re: Jenkins build became unstable: beam_MavenVerify #14

It looks like nowadays, when enabling "Use private Maven repository", you also need to select a strategy. If you want to shield the build against pollution of the local Maven repository by other builds (which I highly recommend, speaking from experience), you should use "Local to the workspace". Th