I believe you can do this today using DynamicAvroDestinations On Mon, Jun 7, 2021 at 11:50 AM Anant Damle <[email protected]> wrote:
> Hi Team, > I am working on a solution with Beam's Java SD, that requires: > > > 1. Read AVRO files using AvroIO.parseGenericRecords > 2. Modify the records (encrypt some attributes) and generate an > updated Schema > 3. Write the modified record with updated Schema > > I'm facing challenges in Step#3, as AvroIO doesn't have an option to read > Schema from a PCollectionView. I found that BigQueryIO.withSchemaFromView > <https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#withSchemaFromView-org.apache.beam.sdk.values.PCollectionView-> > allows > one to provide Schema through a view. > > What should be done for something like AvroIO.withSchemaFromView. > I'm looking for some guidance on how to achieve this. I would be happy to > build and contribute it into the Beam. > > Thanks, > Anant > >
