[Question] Golang: Custom Type Coder

2021-08-01 Thread Tarrence van As
Hi all, Thank you for the great work on Beam. I'm using the golang sdk and hoping someone might be able to help with defining a customer coder for a type. I have a type like this: type Execer struct { Query string `json:"query,omitempty"` Args []interface{} `json:"args,omitempty"` } Where each e

Writing Avro Files to Big Query using Python SDK and Dataflow Runner

2021-08-01 Thread Rajnil Guha
Hi Beam Users, Our pipeline is reading avro files from GCS into Dataflow and writing them into Big Query tables . I am using the WriteToBigQuery transform to write my Pcoll contents into Big Query. My avro file contains about 150-200 fields. We have tested our pipeline by providing the field infor