Re: Streaming inserts BQ with Java SDK Beam

[Andres Angel]

Pablo thanks so much I will explore this method then for   STREAMING_INSERTS
,
this answer worth much for us :)

thanks.

On Tue, May 7, 2019 at 1:05 PM Pablo Estrada  wrote:

> Hi Andres!
> You can definitely do streaming inserts using the Java SDK. This is
> available with BigQueryIO.write(). Specifically, you can use the
> `withMethod`[1] call to specify whether you want batch loads or streaming
> inserts. If you specify streaming inserts, Beam should insert rows as they
> come in bundles.
> Hope that helps
> -P.
>
> [1]
> https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#withMethod-org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.Method-
>
> On Tue, May 7, 2019 at 9:58 AM Andres Angel <
> ingenieroandresan...@gmail.com> wrote:
>
>> Hello everyone,
>>
>> I need to use BigQuery inserts within my beam pipeline, hence I know well
>> the built-in IO options offer `BigQueryIO`, however this will insert in a
>> batch fashion to BQ creating underneath a BQ load job. I instead need to
>> trigger a streaming insert into BQ, and I was reviewing the Java SDK
>> documentation but seems like this is not possible.
>>
>> In the other hand, I have the python SDK and I found this GitHub
>> documentation
>> 
>> code where they are using a method *InsertAll
>> *
>>  which
>> is apparently what I need. If this is official I would like to know if
>> there is a naive fashion to trigger stream inserts in BQ using the Java SDK.
>>
>> thanks so much for your feedback
>> AU
>>
>

Re: Streaming inserts BQ with Java SDK Beam

[Alex Van Boxel]

I think you really need a peculiar reason to force streamingInsert in a
batch job. In batch mode you. Note that you will quickly hit the quota
limit in batch mode: "
Maximum rows per second: 100,000 rows per second, per project", as in batch
load you can process a lot more information in a shorter time.

I know you can force a batch mode in streaming mode, I don't know for the
other way around.

_/
_/ Alex Van Boxel


On Tue, May 7, 2019 at 6:58 PM Andres Angel 
wrote:

> Hello everyone,
>
> I need to use BigQuery inserts within my beam pipeline, hence I know well
> the built-in IO options offer `BigQueryIO`, however this will insert in a
> batch fashion to BQ creating underneath a BQ load job. I instead need to
> trigger a streaming insert into BQ, and I was reviewing the Java SDK
> documentation but seems like this is not possible.
>
> In the other hand, I have the python SDK and I found this GitHub
> documentation
> 
> code where they are using a method *InsertAll
> *
>  which
> is apparently what I need. If this is official I would like to know if
> there is a naive fashion to trigger stream inserts in BQ using the Java SDK.
>
> thanks so much for your feedback
> AU
>

Re: Streaming inserts BQ with Java SDK Beam

[Pablo Estrada]

Hi Andres!
You can definitely do streaming inserts using the Java SDK. This is
available with BigQueryIO.write(). Specifically, you can use the
`withMethod`[1] call to specify whether you want batch loads or streaming
inserts. If you specify streaming inserts, Beam should insert rows as they
come in bundles.
Hope that helps
-P.

[1]
https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#withMethod-org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.Method-

On Tue, May 7, 2019 at 9:58 AM Andres Angel 
wrote:

> Hello everyone,
>
> I need to use BigQuery inserts within my beam pipeline, hence I know well
> the built-in IO options offer `BigQueryIO`, however this will insert in a
> batch fashion to BQ creating underneath a BQ load job. I instead need to
> trigger a streaming insert into BQ, and I was reviewing the Java SDK
> documentation but seems like this is not possible.
>
> In the other hand, I have the python SDK and I found this GitHub
> documentation
> 
> code where they are using a method *InsertAll
> *
>  which
> is apparently what I need. If this is official I would like to know if
> there is a naive fashion to trigger stream inserts in BQ using the Java SDK.
>
> thanks so much for your feedback
> AU
>

Streaming inserts BQ with Java SDK Beam

[Andres Angel]

Hello everyone,

I need to use BigQuery inserts within my beam pipeline, hence I know well
the built-in IO options offer `BigQueryIO`, however this will insert in a
batch fashion to BQ creating underneath a BQ load job. I instead need to
trigger a streaming insert into BQ, and I was reviewing the Java SDK
documentation but seems like this is not possible.

In the other hand, I have the python SDK and I found this GitHub
documentation

code where they are using a method *InsertAll
*
which
is apparently what I need. If this is official I would like to know if
there is a naive fashion to trigger stream inserts in BQ using the Java SDK.

thanks so much for your feedback
AU