Re: Where is the mapreduce runner?

2018-06-19 Thread Reminia Scarlet
@Kenneth When will the runner be released? Does the community have a plan on that ? I've read the second link. The MR runner artifact cant be found in maven central. Maybe it has not beed deployed to maven. On Wed, Jun 20, 2018 at 11:49 AM, Kenneth Knowles wrote: > The MapReduce runner is st

Re: Where is the mapreduce runner?

2018-06-19 Thread Kenneth Knowles
The MapReduce runner is still on a feature branch. See https://beam.apache.org/contribute/#mapreduce-runner We should remove unreleased runners from the capability matrix. May I ask how you arrived at https://beam.apache.org/documentation/runners/mapreduce/ ? Kenn On Tue, Jun 19, 2018 at 8:36 P

Where is the mapreduce runner?

2018-06-19 Thread Reminia Scarlet
I've found that mapreduce runner is supported in beam from below links: https://beam.apache.org/documentation/runners/mapreduce/ https://beam.apache.org/documentation/runners/capability-matrix/ But I cant find the runner on maven.

Re: Returning dataframe from parDo and printing its value - advice?

2018-06-19 Thread OrielResearch Eila Arich-Landkof
Thanks!!! On Mon, Jun 18, 2018 at 4:41 PM, Chamikara Jayalath wrote: > A ParDo should always return an iterator not a string. So if you want to > output a single string it should either be "return [str]" or "yield str". > > > On Mon, Jun 18, 2018 at 1:39 PM OrielResearch Eila Arich-Landkof < > e

Re: Metrics: Non-cumulative values for Distribution

2018-06-19 Thread Jozef Vilcek
I assumed that. Thanks for confirming it. This is what I end up doing, cumulative metrics via Metrics API and metrics with discarding panes custom alongside data. Problem is, that is is not natural and I have to "manually" sync the way I write these to target time series database. I understand tha

Re: Metrics: Non-cumulative values for Distribution

2018-06-19 Thread Etienne Chauchot
Hi Scott and Jozef, Sorry for the late answer, I missed the email. Well, MetricsPusher will aggregate the metrics just as PipelineResult.metrics() does but it will do so at given configurable intervals and export the values. It means that if you configure the export to be every 5s, you will get t