Re: [PROPOSAL] Splittable DoFn - Replacing the Source API with non-monolithic element processing in DoFn

Makes sense. Thanks Eugene. On Mon, Aug 8, 2016, 21:28 Eugene Kirpichov wrote: > Hi Amit, > > Glad you liked the proposal! Yes, adding the power to sequence reading of > sources against other things happening in the pipeline is one of the > biggest benefits. > > I think this proposal is fully co

DoFN Lamdba

Resurrecting a thread from the users list of the same name. I hacked together an example of what this code could look like. I created a modified MapElements

Re: [PROPOSAL] Website page or Jira to host all current proposal discussion and docs

+1 to the overall idea, though I would limit it to large and/or long-term proposals. I like: - JIRA for tracking: that's what it does best. - Google Docs for detailed commenting and revision - basically a wiki with easier commenting - Beam site page for process description and list of current

Re: [PROPOSAL] Splittable DoFn - Replacing the Source API with non-monolithic element processing in DoFn

Hi Amit, Glad you liked the proposal! Yes, adding the power to sequence reading of sources against other things happening in the pipeline is one of the biggest benefits. I think this proposal is fully compatible with having a runner override for TextIO: either way TextIO.Read() produces a composi

Re: [PROPOSAL] Splittable DoFn - Replacing the Source API with non-monolithic element processing in DoFn

Hi Eugene, I really like the proposal, especially the part of embedding a non-Beam job and export jobs prior to pipeline execution - up until now, such work would have been managed by some 3rd party orchestrator that monitors the end of the prepending job, and then executes the pipeline. Having th

Build failed in Jenkins: beam_Release_NightlySnapshot #128

See Changes: [dhalperi] Use EqualsTester in ProxyInvocationHandlerTest -- Started by timer [EnvInject] - Loading node environment variables. Building remotely on beam2 (beam) in works

Build failed in Jenkins: beam_Release_NightlySnapshot #127

See -- Started by timer [EnvInject] - Loading node environment variables. Building remotely on beam2 (beam) in workspace > gi

Re: [PROPOSAL] Website page or Jira to host all current proposal discussion and docs

+1 for the cwiki approach that Aljoshca and Ismael gave examples of. On Mon, Aug 8, 2016 at 2:57 AM, Ismaël Mejía wrote: > +1 for a more formal "Improvement Proposals" with ids we can refer to: > > like Flink does too: > https://cwiki.apache.org/confluence/display/FLINK/ > Flink+Improvement+Prop

Re: [PROPOSAL] Website page or Jira to host all current proposal discussion and docs

+1 for a more formal "Improvement Proposals" with ids we can refer to: like Flink does too: https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals On Mon, Aug 8, 2016 at 10:14 AM, Jean-Baptiste Onofré wrote: > Same think at Karaf: https://cwiki.apache.org/confluence/disp

Re: [PROPOSAL] Splittable DoFn - Replacing the Source API with non-monolithic element processing in DoFn

Jip, thanks, that answers it. On Fri, 5 Aug 2016 at 19:51 Eugene Kirpichov wrote: > Hi Aljoscha, > > AFAIK, the effect of .requiresDeduping() is that the runner inserts a > GBK/dedup transform on top of the read. This seems entirely compatible with > SDF, except it will be decoupled from the SDF

Re: [PROPOSAL] Website page or Jira to host all current proposal discussion and docs

Same think at Karaf: https://cwiki.apache.org/confluence/display/KARAF/ Combine with Jira. Regards JB On 08/08/2016 10:03 AM, Aljoscha Krettek wrote: Please have a look at this: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals We recently started using this proces

Re: [PROPOSAL] Website page or Jira to host all current proposal discussion and docs

Please have a look at this: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals We recently started using this process in Flink and so far are quite happy with it. On Mon, 8 Aug 2016 at 06:52 Jean-Baptiste Onofré wrote: > Good point Ben. > > I would say a "discussion"