Hi Andres! You can definitely do streaming inserts using the Java SDK. This is available with BigQueryIO.write(). Specifically, you can use the `withMethod`[1] call to specify whether you want batch loads or streaming inserts. If you specify streaming inserts, Beam should insert rows as they come in bundles. Hope that helps -P.
[1] https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#withMethod-org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.Method- On Tue, May 7, 2019 at 9:58 AM Andres Angel <ingenieroandresan...@gmail.com> wrote: > Hello everyone, > > I need to use BigQuery inserts within my beam pipeline, hence I know well > the built-in IO options offer `BigQueryIO`, however this will insert in a > batch fashion to BQ creating underneath a BQ load job. I instead need to > trigger a streaming insert into BQ, and I was reviewing the Java SDK > documentation but seems like this is not possible. > > In the other hand, I have the python SDK and I found this GitHub > documentation > <https://beam.apache.org/releases/pydoc/2.4.0/_modules/apache_beam/io/gcp/bigquery.html> > code where they are using a method *InsertAll > <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py>* > which > is apparently what I need. If this is official I would like to know if > there is a naive fashion to trigger stream inserts in BQ using the Java SDK. > > thanks so much for your feedback > AU >
