Re: DynamicMessage in protobufs for re-usable beam pipelines

2020-06-17 Thread amit kumar
Thanks Brian for your response, Alex's code is very helpful. For my current case I will use reflections to get default instance types of proto messages. Another way I think would be to de-couple converters and sinks to bypass this issue and do some of the conversions inside DoFn. Regards, Amit

Re: DynamicMessage in protobufs for re-usable beam pipelines

2020-06-15 Thread Brian Hulette
I don't think I can help with your specific issue, but I can point you to some potentially useful code. +Alex Van Boxel was working on a very similar strategy and added a lot of code for mapping protobufs to Beam schemas which you may be able to take advantage of. He added options to Beam schemas

DynamicMessage in protobufs for re-usable beam pipelines

2020-06-15 Thread amit kumar
Hi, I intend to use Protobuf options to trigger different transforms and use metadata from storage proto options for sink partitioning etc.. and also allow different protobuf message types flowing via the same pipeline, running as different instances of the pipeline. I am able to parse