That would be great, thanks! On Tue, Sep 13, 2022 at 3:00 PM Steve Niemitz <sniem...@apache.org> wrote:
> Ah this is super useful context, thank you! I can submit a couple PRs to > get AvroIO.sink up to parity if that's the way forward. > > On Tue, Sep 13, 2022 at 2:53 PM John Casey via user <user@beam.apache.org> > wrote: > >> Hi Steve, >> >> I've asked around, and it looks like this confusing state is due to a >> migration that isn't complete (and likely won't be until Beam 3.0). >> >> Here is the doc that explains some of the history: >> https://docs.google.com/document/d/1zcF4ZGtq8pxzLZxgD_JMWAouSszIf9LnFANWHKBsZlg/edit >> And a PR that implements some of the changes: >> https://github.com/apache/beam/pull/3817 >> >> Based on this, AvroIO.sink is what we recommend. Please feel free to >> raise issues on Github to account for features you're missing. In addition, >> if you think they are straightforward changes, I'd be happy to discuss >> designs, or look at proposed changes to make these features available. >> >> I hope this helps, >> John >> >> On Mon, Sep 12, 2022 at 3:38 PM Steve Niemitz <sniem...@apache.org> >> wrote: >> >>> We're trying to do some semi-advanced custom logic (custom writers and >>> schemas per destination) with AvroIO, and want to use >>> DynamicAvroDestinations to accomplish this. >>> >>> However, AvroIO.to(DynamicAvroDestinations) is deprecated, but there >>> doesn't seem to be any other way to accomplish what we want here. >>> AvroIO.sink is much less sophisticated than the non-sink options, missing >>> much of the configurability that the non-sink version has. For example, >>> there's no way to project from the UserT -> OutputT with the sink version, >>> only from UserT -> GenericRecord, which isn't what we want. >>> >>> It seems like most things would be trivial to fix or add on the >>> AvroIO.sink implementation, is that the intended way that people would be >>> consuming AvroIO? I'm a little confused with FileIO.write/writeDynamic vs >>> WriteFiles vs AvroIO.write, some seem deprecated, and some seem >>> not-deprecated-but-not-recommended. To add to the confusion AvroIO.write >>> uses WriteFiles, but the documentation for the deprecated >>> AvroIO.to(DynamicAvroDestinations) points to FileIO.write. Which is the >>> "right" one to use? >>> >>
