Re: Writing Avro Files to Big Query using Python SDK and Dataflow Runner

2021-08-09 Thread Pavel Solomin
> wrote a generic BigQuery reader or writer I think, I have seen an example here - https://github.com/the-dagger/dataflow-dynamic-schema/blob/28b7d075c18d6364a67129e56652f452da67a2f6/src/main/java/com/google/cloud/pso/bigquery/BigQuerySchemaMutator.java#L38 This is in Java, but you can try to ada

Writing Avro Files to Big Query using Python SDK and Dataflow Runner

2021-08-01 Thread Rajnil Guha
Hi Beam Users, Our pipeline is reading avro files from GCS into Dataflow and writing them into Big Query tables . I am using the WriteToBigQuery transform to write my Pcoll contents into Big Query. My avro file contains about 150-200 fields. We have tested our pipeline by providing the field infor