Re: Writing Avro Files to Big Query using Python SDK and Dataflow Runner

2021-08-09 Thread Pavel Solomin
> wrote a generic BigQuery reader or writer I think, I have seen an example here - https://github.com/the-dagger/dataflow-dynamic-schema/blob/28b7d075c18d6364a67129e56652f452da67a2f6/src/main/java/com/google/cloud/pso/bigquery/BigQuerySchemaMutator.java#L38 This is in Java, but you can try to ada

Re: Protobuf schema provider row functions break on CamelCased field names

2021-08-09 Thread Chris Hinds
Haha, it probably shouldn’t! I’ll have a look. C. On 9 Aug 2021, at 19:43, Reuven Lax wrote:  I didn't even realize that the proto compiler allowed camel-case field names! I think the only option would be for someone to add support for camel-case proto field names in Beam. On Mon, Aug 9, 2

Protobuf schema provider row functions break on CamelCased field names

2021-08-09 Thread Chris Hinds
Hi, I get an IllegalArgumentException when I call a row function against a proto instance. SerializableFunction myRowFunction = new ProtoMessageSchema().toRowFunction(new TypeDescriptor() {}); MyDataModel.ProtoPayload payload = … Row row = (Row) myRowFunction.apply(payload); It looks like ther