Re: Does ElasticsearchIO in the latest RC support adding document IDs?

Hi, Thanks for the offer, I'd be happy to review your PR. Just wait a bit until I have opened a proper ticket for that. I still need to think more about the design. Among other things, I have to check what ES dev team did for other big data ES IO (es_hadoop) on that particular point. Besides,

Re: Does ElasticsearchIO in the latest RC support adding document IDs?

+1, that is what I had in mind, if I recall correctly this is what es_hadoop connector does. Le 15/11/2017 à 20:22, Tim Robertson a écrit : Hi Chet, I'll be a user of this, so thank you. It seems reasonable although - did you consider letting folk name the document ID field explicitly?  It

Re: Does ElasticsearchIO in the latest RC support adding document IDs?

I think it's the most elegant approach: the user should be able to decide the id field he wants to use. Regards JB On 11/16/2017 09:24 AM, Etienne Chauchot wrote: +1, that is what I had in mind, if I recall correctly this is what es_hadoop connector does. Le 15/11/2017 à 20:22, Tim Robertso

[VOTE] Choose the "new" Spark runner

Hi guys, To illustrate the current discussion about Spark versions support, you can take a look on: -- Spark 1 & Spark 2 Support Branch https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-MODULES This branch contains a Spark runner common module compatible with both Spark 1.x and 2.x. Fo

Re: [VOTE] Choose the "new" Spark runner

[ ] Use Spark 1 & Spark 2 Support Branch [X] Use Spark 2 Only Branch On 16 November 2017 at 15:08, Jean-Baptiste Onofré wrote: > Hi guys, > > To illustrate the current discussion about Spark versions support, you can > take a look on: > > -- > Spark 1 & Spark 2 Support Branch > > https:/

Re: Does ElasticsearchIO in the latest RC support adding document IDs?

I'd add to the idea here with the A solution. What about also supporting a user function to provide the ID given the record. I say this because I'm starting to also look into how to get the ESIO writer to support dynamic index based on information contained within the event. For which just looking

Re: Does ElasticsearchIO in the latest RC support adding document IDs?

Chet, FYI, here is the ticket and the design proposal: https://issues.apache.org/jira/browse/BEAM-3201. If you still want to code that improvement, give me your jira id and I will assign the ticket to you. Otherwise I can code it as well. Best Etienne Le 16/11/2017 à 09:19, Etienne Chauch

Re: Does ElasticsearchIO in the latest RC support adding document IDs?

Sure, I’d be happy to take it on. My JIRA ID is chetaldrich. We can continue discussion on that ticket. Chet > On Nov 16, 2017, at 7:57 AM, Etienne Chauchot wrote: > > Chet, > FYI, here is the ticket and the design proposal: > https://issues.apache.org/jira/browse/BEAM-3201 >

@DoFn.Setup not called

This one is weird. A DoFn I wrote: - stateful - used plenty in a streaming pipeline - direct and dataflow runners - works fine Now: - new batch pipeline - @DoFn.Setup method not called - direct runner works properly (logs from setup method are output) - dataflow runner simply doesn't call the set

Re: [VOTE] Choose the "new" Spark runner

[ ] Use Spark 1 & Spark 2 Support Branch [X] Use Spark 2 Only Branch Spark 2 has been out for a while, so probably not going to offend many people. Jacob On Thu, Nov 16, 2017 at 5:45 AM, Neville Dipale wrote: > [ ] Use Spark 1 & Spark 2 Support Branch > [X] Use Spark 2 Only Branch > >

Re: Slack Channel

Me too, if you don't mind. Jacob On Thu, Nov 9, 2017 at 2:09 PM, Lukasz Cwik wrote: > Invite sent, welcome. > > On Thu, Nov 9, 2017 at 2:08 PM, Fred Tsang wrote: > >> Hi, >> >> Please add me to the slack channel. >> >> Thanks, >> Fred >> >> Ps. I think "BeamTV" would be a great YouTube channel

Re: Slack Channel

If you wouldn’t mind I’d like an invite as well. Chet > On Nov 16, 2017, at 4:58 PM, Jacob Marble wrote: > > Me too, if you don't mind. > > Jacob > > On Thu, Nov 9, 2017 at 2:09 PM, Lukasz Cwik > wrote: > Invite sent, welcome. > > On Thu, Nov 9, 2017 at 2:08 PM, Fr

Re: Slack Channel

Invites have been sent to you Jacob and Chet, please check your inboxes. On Thu, Nov 16, 2017 at 5:42 PM, Chet Aldrich wrote: > If you wouldn’t mind I’d like an invite as well. > > Chet > > On Nov 16, 2017, at 4:58 PM, Jacob Marble wrote: > > Me too, if you don't mind. > > Jacob > > On Thu, Nov

Re: @DoFn.Setup not called

​I've been using DoFn.Setup method in Dataflow and it seems to be working fine.​ On Thu, Nov 16, 2017 at 4:56 PM, Jacob Marble wrote: > This one is weird. > > A DoFn I wrote: > - stateful > - used plenty in a streaming pipeline > - direct and dataflow runners > - works fine > > Now: > - new batc

Re: @DoFn.Setup not called

Could you give more details, e.g. a code snippet that reproduces the issue, and describe how you determine that @Setup hasn't been called? On Thu, Nov 16, 2017 at 6:58 PM Derek Hao Hu wrote: > ​I've been using DoFn.Setup method in Dataflow and it seems to be working > fine.​ > > On Thu, Nov 16,