Re: Third Apache Beam meet up in London

2018-03-01 Thread Griselda Cuevas
Great and thanks for organizing guys! On 27 February 2018 at 03:22, Matthias Baetens wrote: > Hi all, > > We are very excited to announce the third Apache Beam meetup in London *the > 5th of March*. > > Tyler Akidau (Google) will be talking about Streaming SQL, while Victor > Kotai (Qubit) wi

Re: Running code before pipeline starts

2018-03-01 Thread Andrew Jones
Thanks Lukasz, went with the side input approach and it worked perfectly! On Wed, 28 Feb 2018, at 18:28, Lukasz Cwik wrote: > You should use a side input and not an empty PCollection that you > flatten.> > Since > ReadA --> Flatten --> ParDo > ReadB -/ > can be equivalently executed as: > ReadA -

Error running 2.3.0 on Dataflow

2018-03-01 Thread Andrew Jones
Hi, I've tried to upgrade a Beam job to 2.3.0 and deploy on Dataflow and getting the following error: 2018-03-01 10:52:35 INFO PackageUtil:316 - Uploading 169 files from PipelineOptions.filesToStage to staging location to prepare for execution. Exception in thread "main" java.lang.RuntimeExcep

Re: BigQueryIO streaming inserts - poor performance with multiple tables

2018-03-01 Thread Josh
Hi Cham, Thanks, I have emailed the dataflow-feedback email address with the details. Best regards, Josh On Thu, Mar 1, 2018 at 12:26 AM, Chamikara Jayalath wrote: > Could be a DataflowRunner specific issue. Would you mind reporting this > with corresponding Dataflow job IDs to either Dataflow