Re: Batch loads in streaming pipeline - withNumFileShards

2017-11-15 Thread Lukasz Cwik
Filed https://issues.apache.org/jira/browse/BEAM-3198 for the IllegalArgumentException Do you mind posting a little code snippet of how you build the BQ IO connector on BEAM-3198? On Wed, Nov 15, 2017 at 12:18 PM, Arpan Jain wrote: > Hi, > > I am trying to use Method.FILE_LOADS for loading da

Batch loads in streaming pipeline - withNumFileShards

2017-11-15 Thread Arpan Jain
Hi, I am trying to use Method.FILE_LOADS for loading data into BQ in my streaming pipeline using RC3 release of 2.2.2. It looks like withNumFileShards needs to be also set for using this. Couple of questions regarding this: * Any guidelines on what's a good value for this? FWIW my pipeline is goi