Re: Support for avro when writing to BQ using the streaming write API

2021-08-05 Thread Steve Niemitz
Yeah, it was supported for FILE_LOADS only, which was the only thing we ever needed to use :) (streaming inserts are too expensive and unreliable to be useful) Avro (GenericRecord) -> Row does use an intermediate object (it builds a Row), it doesn't wrap the record like SpecificRecord does sadly.

Support for avro when writing to BQ using the streaming write API

2021-08-05 Thread Steve Niemitz
We currently have a lot of jobs that write to BQ using avro (using withAvroFormatFunction), and would like to start experimenting with the streaming write API. I see two options for how to do this: - Implement support for avro -> protobuf like TableRow and beam Row do - Add something like withProt