Re: [PROPOSAL] "Requires deterministic input"

On Mon, Dec 11, 2017 at 8:47 PM, Kenneth Knowles wrote: > In discussion on https://github.com/apache/beam/pull/4135 and offline it > came up how this should interact with stateful/timely DoFns. > > Highlights: > > - It was suggested to support it for @OnTimer which would mean

Re: [PROPOSAL] "Requires deterministic input"

In discussion on https://github.com/apache/beam/pull/4135 and offline it came up how this should interact with stateful/timely DoFns. Highlights: - It was suggested to support it for @OnTimer which would mean that the state is stable on retry - But an unfortunate corollary is that for stateful

Re: A question regarding BEAM-3280

+1. For example, one might do something like split_lines_result = (lines | beam.ParDo(SplitLinesToWordsFn().with_output_types( beam.typehints.Generator[int],

Re: A question regarding BEAM-3280

Hi Norio, Thank you for your interest. If you would like to work on this I can assign the JIRA to you. I do not think this change in sufficient or correct. This reads as if SplitLinesToWordsFn returns a Tuple of things, however instead it produces three unrelated collections of different types.

Re: [DISCUSS] [Java] Private shaded dependency uber jars

I would suggest that either we use: 1) A common deps package containing shaded dependencies allows for Pros * doesn't require the user to build an uber jar Risks * dependencies package will keep growing even if something is or isn't needed by all of Apache Beam leading to a large jar anyways

Re: [POLL] Dropping support for Java 7 in Beam 2.3

Intermediate update: right now it's 219 votes on Twitter (and 0 on this thread: seems that people find Twitter a more convenient medium!), with 96% who are either on Java 8 or can easily switch, vs. 4% who can not easily switch. Using a proportion confidence interval calculator, so far we're

Re: [DISCUSS] [Java] Private shaded dependency uber jars

Thanks for bringing that back. Indeed guava is shaded in different uber-jar. Maybe we can have a common deps module that we include once (but the user will have to explicitly define the dep) ? Basically, what do you propose for protobuf (unfortunately, I don't see an obvious) ? Regards JB

Re: [DISCUSSION] Runner agnostic metrics extractor?

Hi, thanks for the doc. I left some comments. Regards JB On 12/11/2017 05:33 PM, Etienne Chauchot wrote: Hi all, I sketched a little doc [1] about this subject. It tries to sum up the differences between the runners towards metrics extraction and propose some possible designs to have a

Re: [DISCUSS] [Java] Private shaded dependency uber jars

Hello, I wanted to bring back this subject because I think we should take action on this and at least first have a shaded version of guava. I was playing with a toy project and I did the procedure we use to submit jars to a Hadoop cluster via Flink/Spark which involves creating an uber jar and I

Re: [DISCUSSION] Runner agnostic metrics extractor?

Hi all, I sketched a little doc [1] about this subject. It tries to sum up the differences between the runners towards metrics extraction and propose some possible designs to have a runner agnostic extraction of the metrics. It is a 2 pages long doc, can you please comment it, and correct it

reduced availability lately and until the end of january

Hi guys, These last weeks I've been off one to two days a week, and I'll be off that way until the end of January. I just wanted to let you know about this availability decrease. Etienne