Re: Batch loads in streaming pipeline - withNumFileShards

[Lukasz Cwik]

Filed https://issues.apache.org/jira/browse/BEAM-3198 for the
IllegalArgumentException
Do you mind posting a little code snippet of how you build the BQ IO
connector on BEAM-3198?




On Wed, Nov 15, 2017 at 12:18 PM, Arpan Jain  wrote:

> Hi,
>
> I am trying to use Method.FILE_LOADS for loading data into BQ in my
> streaming pipeline using RC3 release of 2.2.2. It looks
> like withNumFileShards needs to be also set for using this. Couple of
> questions regarding this:
>
> * Any guidelines on what's a good value for this? FWIW my pipeline is
> going to write > 100K messages/second to around 500 tables
> using DynamicDestinations. I am trying to set a triggering frequency of 30
> minutes or 1 hour.
>
> * Can we throw a better error message when it is not set? The exception
> thrown by the pipeline if this is not set is "Exception in thread "main"
> java.lang.IllegalArgumentException" without any message.
>

Batch loads in streaming pipeline - withNumFileShards

[Arpan Jain]

Hi,

I am trying to use Method.FILE_LOADS for loading data into BQ in my
streaming pipeline using RC3 release of 2.2.2. It looks
like withNumFileShards needs to be also set for using this. Couple of
questions regarding this:

* Any guidelines on what's a good value for this? FWIW my pipeline is going
to write > 100K messages/second to around 500 tables
using DynamicDestinations. I am trying to set a triggering frequency of 30
minutes or 1 hour.

* Can we throw a better error message when it is not set? The exception
thrown by the pipeline if this is not set is "Exception in thread "main"
java.lang.IllegalArgumentException" without any message.