Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

PST mornings are better, because they are evening/nights for me. Friday would work-out best for me. On Mon, Mar 13, 2017 at 11:46 PM Eugene Kirpichov wrote: > Awesome!!! > > Amit - remind me your time zone? JB, do you want to join? > I'm free this week all afternoons (say after 2pm) in Pacific T

Re: Batch loading for streaming pipelines

This seems like a good topic for user@ so I've moved it there (dev@ to BCC). You can get a bounded PCollection from KafkaIO via either of .withMaxNumRecords() or .withMaxReadTime(). Whether or not that will meet your use case would depend on more details of what you are computing. Periodic batch

Batch loading for streaming pipelines

Hi, We run multiple streaming pipelines using cloud dataflow that read from Kafka and write to BigQuery. We don't mind a few hours delay and are thinking of avoiding the costs associated with streaming data into BigQuery. Is there already a support (or a future plan) for such a scenario? If not

Re: [VOTE] Release 0.6.0, release candidate #2

Conclusion (see JIRA): Not a release blocker (but still a bug in TestPipeline). On Mon, Mar 13, 2017 at 5:40 PM Eugene Kirpichov wrote: > +Aljoscha Krettek > > On Mon, Mar 13, 2017 at 5:30 PM Eugene Kirpichov > wrote: > > +Stas Levin +Thomas Groh > > On Mon, Mar 13, 2017 at 5:30 PM Eugene Ki

Re: [VOTE] Release 0.6.0, release candidate #2

+Aljoscha Krettek On Mon, Mar 13, 2017 at 5:30 PM Eugene Kirpichov wrote: > +Stas Levin +Thomas Groh > > On Mon, Mar 13, 2017 at 5:30 PM Eugene Kirpichov > wrote: > > https://issues.apache.org/jira/browse/BEAM-1712 might be a release > blocker. > > On Mon, Mar 13, 2017 at 4:53 PM Ahmet Altay

Re: [VOTE] Release 0.6.0, release candidate #2

+Stas Levin +Thomas Groh On Mon, Mar 13, 2017 at 5:30 PM Eugene Kirpichov wrote: > https://issues.apache.org/jira/browse/BEAM-1712 might be a release > blocker. > > On Mon, Mar 13, 2017 at 4:53 PM Ahmet Altay > wrote: > > Thank you for all the comment so far. > > On Mon, Mar 13, 2017 at 4:23

Re: [VOTE] Release 0.6.0, release candidate #2

https://issues.apache.org/jira/browse/BEAM-1712 might be a release blocker. On Mon, Mar 13, 2017 at 4:53 PM Ahmet Altay wrote: > Thank you for all the comment so far. > > On Mon, Mar 13, 2017 at 4:23 PM, Ted Yu wrote: > > > bq. I would prefer that we have a .tar.gz release > > > > +1 > > > > O

Re: [VOTE] Release 0.6.0, release candidate #2

Thank you for all the comment so far. On Mon, Mar 13, 2017 at 4:23 PM, Ted Yu wrote: > bq. I would prefer that we have a .tar.gz release > > +1 > > On Mon, Mar 13, 2017 at 4:21 PM, Ismaël Mejía wrote: > > > ​+1 (non-binding) > > > > - verified signatures + checksums > > - run mvn clean install

Re: [VOTE] Release 0.6.0, release candidate #2

bq. I would prefer that we have a .tar.gz release +1 On Mon, Mar 13, 2017 at 4:21 PM, Ismaël Mejía wrote: > ​+1 (non-binding) > > - verified signatures + checksums > - run mvn clean install -Prelease, all artifacts build and the tests run > smoothly (modulo some local issues I had with the ins

Re: [VOTE] Release 0.6.0, release candidate #2

​+1 (non-binding) - verified signatures + checksums - run mvn clean install -Prelease, all artifacts build and the tests run smoothly (modulo some local issues I had with the installation of tox for the python sdk, I created a PR to fix those in case other people can have the same trouble). Some

Build failed in Jenkins: beam_SeedJob #159

See -- GitHub pull request #2204 of commit dcae644305d9cc24a94fc87f515faffd5af72f10, no merge conflicts. Setting status of dcae644305d9cc24a94fc87f515faffd5af72f10 to PENDING with url https:

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Awesome!!! Amit - remind me your time zone? JB, do you want to join? I'm free this week all afternoons (say after 2pm) in Pacific Time, and mornings of Wed & Fri. We'll probably need half an hour to an hour. On Mon, Mar 13, 2017 at 1:29 PM Aljoscha Krettek wrote: > I whipped up a quick version

Re: [VOTE] Release 0.6.0, release candidate #2

+1 (binding) On Mon, Mar 13, 2017 at 11:10 AM, Robert Bradshaw wrote: > On Sat, Mar 11, 2017 at 11:19 PM, Ahmet Altay > wrote: > >> On Sat, Mar 11, 2017 at 11:48 AM, Robert Bradshaw < >> rober...@google.com.invalid> wrote: >> >> > On Fri, Mar 10, 2017 at 9:05 PM, Ahmet Altay >> > wrote: >> > >

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

I whipped up a quick version for Flink that seems to work: https://github.com/apache/beam/pull/2235 There are still two failing tests, as described in the PR. On Mon, Mar 13, 2017, at 20:10, Amit Sela wrote: > +1 for a video call. I think it should be pretty straight forward for the > Spark runn

Re: Style: how much testing for transform builder classes?

+0.5 My two cents, * However trivial the test is it should be added unless user has a easy workaround to not having to wait for a few days until the trivial fixes are merged to beam and then propagated to the runner. * While I agree with trivial tests like "ensuring meaningful error message

Re: [VOTE] Release 0.6.0, release candidate #2

bq. That name makes sense to me +1 Maybe change the subject of this thread and send to dev@ to raise awareness ? On Mon, Mar 13, 2017 at 11:10 AM, Robert Bradshaw < rober...@google.com.invalid> wrote: > On Sat, Mar 11, 2017 at 11:19 PM, Ahmet Altay > wrote: > > > On Sat, Mar 11, 2017 at 11:48

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

+1 for a video call. I think it should be pretty straight forward for the Spark runner after the work on read from UnboundedSource and after GroupAlsoByWindow, but from my experience such a call could move us forward fast enough. On Mon, Mar 13, 2017, 20:37 Eugene Kirpichov wrote: > Hi all, > >

Re: [VOTE] Release 0.6.0, release candidate #2

On Sat, Mar 11, 2017 at 11:19 PM, Ahmet Altay wrote: > On Sat, Mar 11, 2017 at 11:48 AM, Robert Bradshaw < > rober...@google.com.invalid> wrote: > > > On Fri, Mar 10, 2017 at 9:05 PM, Ahmet Altay > > wrote: > > > > > Hi everyone, > > > > > > Please review and vote on the release candidate #2 for

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi all, Let us continue working on this. I am back from various travels and am eager to help. Amit, JB - would you like to perhaps have a videocall to hash this out for the Spark runner? Aljoscha - are the necessary Flink changes done / or is the need for them obviated by using the (existing) ru

Re: Beam deploy uploads some artifacts twice

I haven't seen this specific issue personally, but, generally speaking, multiple executions are often caused by an incorrect "execution id" in a pom.xml. Instead of re-configuring the default execution, it creates a new execution -- and this has previously caused issues elsewhere. Usually, this ca

Jenkins build is back to stable : beam_Release_NightlySnapshot #355

See