Re: Low availability on my end in the coming 3 weeks

+1 to good travels & move. Your absence will surely be noticed for its own sake. See you when you resurface :-) Kenn On Wed, Apr 12, 2017 at 6:28 AM, Jean-Baptiste Onofré wrote: > Thanks for the update and your trust Amit ! > > Safe travel and take your time to move to your

Re: Naming of Combine.Globally

On Tue, Apr 18, 2017 at 3:03 AM, Wesley Tanaka wrote: > I believe that foldl in Haskell https://www.haskell.org/hoogle/?hoogle=foldl > admits a separate accumulator type from the type of the data structure being > "folded" > And, well, python lets you have your way

Re: [DISCUSSION] PAssert success/failure count validation for all runners

So to summarize, most seem to agree that: 1) We should verify PAssert execution occurred in all runners. 2) We should verify this using metrics in sdk-java-core for runners which support metrics. This will save those runner writers from having to verify this in the runner code. See:

Re: Pipeline termination in the unified Beam model

BEAM-593 is blocked by Flink issues: - https://issues.apache.org/jira/browse/FLINK-2313: Change Streaming Driver Execution Model - https://issues.apache.org/jira/browse/FLINK-4272: Create a JobClient for job control and monitoring where the second is kind of a duplicate of the first one.

Re: Build failed in Jenkins: beam_SeedJob #214

Yup -- it looks like we're going to need reapproval when we change our jobs[1]. [1] https://github.com/jenkinsci/job-dsl-plugin/wiki/Script-Security#script-approval On Tue, Apr 18, 2017 at 10:20 AM, Ted Yu wrote: > Thanks Jason for the effort. > Looks like we hit this: > >

Re: Build failed in Jenkins: beam_SeedJob #214

Thanks Jason for the effort. Looks like we hit this: ERROR: script not yet approved for use On Tue, Apr 18, 2017 at 10:16 AM, Jason Kuster < jasonkus...@google.com.invalid> wrote: > I'm looking into this currently as well; that's one of the mitigations I'm > considering too but I'm giving the

Re: Build failed in Jenkins: beam_SeedJob #214

I'm looking into this currently as well; that's one of the mitigations I'm considering too but I'm giving the evaluate thing a try[1][2] (once it starts running -- executors are full currently). [1] https://builds.apache.org/view/Beam/job/beam_SeedJob/215/ [2]

Re: Build failed in Jenkins: beam_SeedJob #214

To unblock the builds, how about embedding functions used by respective scripts in the scripts themselves ? e.g. buildPerformanceTest is only used by the following scripts: .test-infra/jenkins/job_beam_PerformanceTests_Dataflow.groovy: common_job_properties.buildPerformanceTest(delegate,

Re: Build failed in Jenkins: beam_SeedJob #214

Not so simple, unfortunately [1]. Ideas welcome ;-) Davor [1] https://github.com/jenkinsci/job-dsl-plugin/wiki/Migration#migrating-to-160 On Tue, Apr 18, 2017 at 9:57 AM, Ted Yu wrote: > I wonder if we should adopt the suggestion here (involving evaluate): >

Re: Read/Write Transform Documentation

Hi Andrew, I'm excited to hear you're working on an I/O - I'd love to hear any feedback about the docs we've got written so far. Sorry they're in a partially completed state. Are you looking to develop in python or java? There's more specific docs for python available in the python SDK guide

Read/Write Transform Documentation

Good Morning Beam Devs! I was looking through the beam dev docs and saw a TODO in the authoring IO section. I was wondering if this had been completed or is located elsewhere on the website. (https://beam.apache.org/documentation/io/authoring-overview/) I am working on making IO for Apache

Re: Naming of Combine.Globally

...Curiously enough, ReduceFn is by far the closest of all these to a sequential fold. It is also internal (runner-facing rather than user-facing). On Tue, Apr 18, 2017 at 8:27 AM Dan Halperin wrote: > Great discussion! As Aljoscha says, Fold, Reduce, and Combine

Re: Naming of Combine.Globally

Great discussion! As Aljoscha says, Fold, Reduce, and Combine are all intertwined and not quite identical as we use them. Another simple but perhaps coy answer is that if you read the MapReduce paper by Dean and Ghemawat that started this all, they used "Map", "Reduce", and "Combine" (see section

Re: Pipeline termination in the unified Beam model

Ted, the timeout is needed mostly for testing purposes. AFAIK there is no easy way to express the fact a source is "done" in a Spark native streaming application. Moreover, the Spark streaming "native" flow can either "awaitTermination()" or "awaitTerminationOrTimeout(...)". If you

Re: Pipeline termination in the unified Beam model

Why is the timeout needed for Spark ? Thanks > On Apr 18, 2017, at 3:05 AM, Etienne Chauchot wrote: > > +1 on "runners really terminate in a timely manner to easily programmatically > orchestrate Beam pipelines in a portable way, you do need to know whether > the pipeline

Re: Naming of Combine.Globally

I believe that foldl in Haskell https://www.haskell.org/hoogle/?hoogle=foldl  admits a separate accumulator type from the type of the data structure being "folded" And, well, python lets you have your way with mixing types, but this certainly works as another example:python -c

Re: Pipeline termination in the unified Beam model

+1 on "runners really terminate in a timely manner to easily programmatically orchestrate Beam pipelines in a portable way, you do need to know whether the pipeline will finish without thinking about the specific runner and its options" As an example, in Nexmark, we have streaming mode tests,

Re: Naming of Combine.Globally

Hi, I think both fold and reduce fail to capture all the power or (what we call) combine. Reduce requires a function of type (T, T) -> T. It requires that the output type be the same as the input type. Fold takes a function (T, A) -> A where T is the input type and A is the accumulation type.

Jenkins build is still unstable: beam_Release_NightlySnapshot #392

See

Build failed in Jenkins: beam_SeedJob #213

See Changes: [altay] Add no-else return to pylintrc [chamikara] Update assertions of source_test_utils from camelcase to [dhalperi] Only compile HIFIO ITs when compiling with java 8. [tgroh] Set the Project of a