Re: Beam Docs Contributor

Welcome Rose ! Etienne Le lundi 30 juillet 2018 à 10:10 -0700, Thomas Weise a écrit : > Welcome Rose, and looking forward to the docs update! > On Mon, Jul 30, 2018 at 9:15 AM Henning Rohde wrote: > > Welcome Rose! Great to have you here. > > On Mon, Jul 30, 2018 at 2:23 AM Ismaël Mejía wrote: >

Re: Beam Docs Contributor

That sounds great, Rose. Welcome! On 21.08.18 09:21, Etienne Chauchot wrote: > Welcome Rose ! > > Etienne > > Le lundi 30 juillet 2018 à 10:10 -0700, Thomas Weise a écrit : >> Welcome Rose, and looking forward to the docs update! >> >> On Mon, Jul 30, 2018 at 9:15 AM Henning Rohde >

Re: dulicate key-value elements lost when transfering them as side-inputs

Hi, looking for reviewer https://github.com/apache/beam/pull/6257 And maybe some help with failing test in mqtt IO (timeout). Vaclav From: Lukasz Cwik Sent: Monday, August 20, 2018 6:12:24 PM To: dev Subject: Re: dulicate key-value elements lost when transfer

Re: dulicate key-value elements lost when transfering them as side-inputs

Thanks for this Vaclav The failing test (1 minute timeout exception) is something we see sometimes and indicates issues in the build environment or a flakey test. I triggered another build by leaving a comment in the PR - just fyi, this is something you can also do in the future. On Tue, Au

Re: Beam Docs Contributor

Welcome Rose!!! On Tue, Aug 21, 2018 at 12:57 AM Maximilian Michels wrote: > That sounds great, Rose. Welcome! > > On 21.08.18 09:21, Etienne Chauchot wrote: > > Welcome Rose ! > > > > Etienne > > > > Le lundi 30 juillet 2018 à 10:10 -0700, Thomas Weise a écrit : > >> Welcome Rose, and looking f

Re: Bootstrapping Beam's Job Server

Thanks Henning and Thomas. It looks like a) we want to keep the Docker Job Server Docker container and rely on spinning up "sibling" SDK harness containers via the Docker socket. This should require little changes to the Runner code. b) have the InProcess SDK harness as an alternative way to r

Re: Process JobBundleFactory for portable runner

For reference, here is corresponding JIRA issue for this thread: https://issues.apache.org/jira/browse/BEAM-5187 On 16.08.18 11:15, Maximilian Michels wrote: Makes sense to have an option to run the SDK harness in a non-dockerized environment. I'm in the process of creating a Docker entry poin

Re: Bootstrapping Beam's Job Server

It is also worth to mention that apart of the testing/development use case there is also the case of supporting people running in Hadoop distributions. There are two extra reasons to want a process based version: (1) Some Hadoop distributions run in machines with really old kernels where docker sup

Beam Summit London 2018

Hi everyone, We are happy to invite you to the first Beam Summit in London. The summit will be held in London at Level39 on *October 1 and 2.* You can register to attend for free on the Eventbrite page

Re: Beam Summit London 2018

Hi Matthias, we (Peat / Plantix) might be interested by submitting a talk and I would like to know if we can get access to the list of already submitted "Title" to avoid submitting on similar topic! Cheers, Pascal On Tue, Aug 21, 2018 at 1:59 PM, Matthias Baetens wrote: > Hi everyone, > > We are

Re: Process JobBundleFactory for portable runner

One thing to consider that we've talked about in the past. It might make sense to extend the environment proto and have the SDK be explicit about which kinds of environment it supports: https://github.com/apache/beam/blob/8c4f4babc0b0d55e7bddefa3f9f9ba65d21ef139/model/pipeline/src/main/proto/beam

Re: Beam Summit London 2018

Hi, What'd be the duration of the talks? So I can scope the contents of my proposal. Looking forward to the summit! J On Tue, 21 Aug 2018, 14:47 Pascal Gula, wrote: > Hi Matthias, > we (Peat / Plantix) might be interested by submitting a talk and I would > like to know if we can get access to

Re: Process JobBundleFactory for portable runner

I would model the environment to be more free form then enums such that we have forward looking extensibility and would suggest to follow the same pattern we use on PTransforms (using an URN and a URN specific payload). Note that in this case we may want to support a list of supported environments

Re: Travis apache credentials

I was imagining the signing itself would still be manual. (Frankly, I would feel odd having travis sign them for me...) On Mon, Aug 20, 2018 at 10:20 PM Lukasz Cwik wrote: > > If you can't get an answer quickly, its best to read the Apache policy on > release signing: http://www.apache.org/dev/re

Re: Bug or confusing python code? Are these the same element count metrics?

On Tue, Aug 21, 2018 at 2:05 AM Alex Amato wrote: > > I discovered something while trying to update test_progress_metrics in > fn_api_runner_tests.py to inspect the returned MonitoringInfos in addition to > the already returned metrics format. > > This metric appears to be added twice using the

Re: Process JobBundleFactory for portable runner

By "enum" in quotes, I meant the usual open URN style pattern not an actual enum. Sorry if that wasn't clear. On Tue, Aug 21, 2018 at 11:51 AM Lukasz Cwik wrote: > I would model the environment to be more free form then enums such that we > have forward looking extensibility and would suggest to

Re: Beam application upgrade on Flink crashes

Flink 1.7 will change the way the "restore serializer" is handled, which should make it much easier to handle such cases. Especially breaking java class version format will not be an issue anymore. That should help to make it easier to give the Beam-on-Flink runner cross version compatibility. O

Re: Discussion: Scheduling across runner and SDKHarness in Portability framework

I think it will be useful to the runner to know upfront what the fundamental threading capabilities are for the SDK harness (say, "fixed", "linear", "dynamic", ..) so that the runner can upfront make a good static decision on #harnesses and how many resources they should each have. It's wasteful to

Re: Bootstrapping Beam's Job Server

I believe supporting a simple Process environment makes sense. It would be best if we didn't make the Process route solve all the problems that Docker solves for us. In my opinion we should limit the Process route to assume that the execution environment: * has all dependencies and libraries instal

Re: Discussion: Scheduling across runner and SDKHarness in Portability framework

Henning, can you clarify by what you mean with send non-executable bundles to the SDK harness and how it is useful for Flink? On Tue, Aug 21, 2018 at 2:01 PM Henning Rohde wrote: > I think it will be useful to the runner to know upfront what the > fundamental threading capabilities are for the S

Re: Beam Summit London 2018

Hi there, We'll have 20min talks with 10min for Q&A, so 30min total. G On Tue, 21 Aug 2018 at 11:45, javier ramirez < javier.ramirez.gom...@gmail.com> wrote: > Hi, > > What'd be the duration of the talks? So I can scope the contents of my > proposal. > > Looking forward to the summit! > > J

Re: Bootstrapping Beam's Job Server

Agree with Luke. Perhaps something simple, prescriptive yet flexible, such as custom command line (defined in the environment proto) rooted at the base of the provided artifacts and either passed the same arguments or defined in the container contract or made available through substitution. That wa

Re: Discussion: Scheduling across runner and SDKHarness in Portability framework

Sending bundles that cannot be executed, i.e., the situation described to cause deadlock in Flink in the beginning of the thread with mapB. The discussion of exposing (or assuming an infinitely large) concurrency level -- while a useful concept in its own right -- came around as a way to unblock ma