On Tue, Nov 15, 2022 at 11:50 AM Damon Douglas via dev <dev@beam.apache.org> wrote:
> Hello Everyone, > > Do we like the following Java class naming convention for > SchemaTransformProviders [1]? The proposal is: > > <IOName>(Read|Write)SchemaTransformProvider > > > *For those new to Beam, even if this is your first day, consider > yourselves a welcome contributor to this conversation. Below are > definitions/references and a suggested learning guide to understand this > email.* > > Explanation > > The <IOName> identifies the Beam I/O [2] and Read or Write identifies a > read or write Ptransform, respectively. > Schema-aware transforms are not restricted to I/Os. An arbitrary transform can be a Schema-Transform. Also, designation Read/Write does not map to an arbitrary transform. Probably we should try to make this more generic ? Also, probably what's more important is the identifier of the SchemaTransformProvider being unique. Note the class name (the latter is guaranteed to be unique if we follow the Java package naming guidelines). FWIW, we came up with a similar generic URN naming scheme for cross-language transforms: https://beam.apache.org/documentation/programming-guide/#1314-defining-a-urn Thanks, Cham > For example, to implement a SchemaTransformProvider [1] for > BigQueryIO.Write[7], would look like: > > BigQueryWriteSchemaTransformProvider > > > And to implement a SchemaTransformProvider for PubSubIO.Read[8] would like > like: > > PubsubReadSchemaTransformProvider > > > Definitions/References > > [1] *SchemaTransformProvider*: A way for us to instantiate Beam IO > transforms using a language agnostic configuration. > SchemaTransformProvider builds a SchemaTransform[3] from a Beam Row[4] that > functions as the configuration of that SchemaProvider. > > https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/schemas/transforms/SchemaTransformProvider.html > > [2] *Beam I/O*: PTransform for reading from or writing to sources and > sinks. > https://beam.apache.org/documentation/programming-guide/#pipeline-io > > [3] *SchemaTransform*: An interface containing a buildTransform method > that returns a PCollectionRowTuple[5] to PCollectionRowTuple PTransform. > > https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/schemas/transforms/SchemaTransform.html > > [4] *Row*: A Beam Row is a generic element of data whose properties are > defined by a Schema[5]. > > https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/values/Row.html > > [5] *Schema*: A description of expected field names and their data types. > > https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/schemas/Schema.html > > [6] *PCollectionRowTuple*: A grouping of Beam Rows[4] into a single > PInput or POutput tagged by a String name. > > https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/values/PCollectionRowTuple.html > > [7] *BigQueryIO.Write*: A PTransform for writing Beam elements to a > BigQuery table. > > https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html > > [8] *PubSubIO.Read*: A PTransform for reading from Pub/Sub and emitting > message payloads into a PCollection. > > https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.Read.html > > Suggested Learning/Reading to understand this email > > 1. https://beam.apache.org/documentation/programming-guide/#overview > 2. https://beam.apache.org/documentation/programming-guide/#transforms > (Up to 4.1) > 3. https://beam.apache.org/documentation/programming-guide/#pipeline-io > 4. https://beam.apache.org/documentation/programming-guide/#schemas >
