[ https://issues.apache.org/jira/browse/BEAM-9035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Van Boxel updated BEAM-9035: --------------------------------- Parent: BEAM-9275 Issue Type: Sub-task (was: Task) > Typed options for Row Schema and Fields > --------------------------------------- > > Key: BEAM-9035 > URL: https://issues.apache.org/jira/browse/BEAM-9035 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core > Reporter: Alex Van Boxel > Assignee: Alex Van Boxel > Priority: Major > Fix For: 2.19.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > This is the first issue of a multipart commit: this ticket implements the > basic infrastructure of options on row and field. > Full explanation: > Introduce the concept of Options in Beam Schema’s to add extra context to > fields and schema. In contracts to metadata, options would be added to > fields, logical types and rows. In the options schema convertors can add > options/annotations/decorators that were in the original schema, this context > can be used in the rest of the pipeline for specific transformations or > augment the end schema in the target output. > Examples of options are: > * informational: like the source of the data, ... > * drive decisions further in the pipeline: flatten a row into another, > rename a field, ... > * influence something in the output: like cluster index, primary key, ... > * logical type information > And option is a key/typed value combination. The advantages of having the > value types is: > * Having strongly typed options would give a *portable way of Logical Types* > to have structured information that could be shared over different languages. > * This could keep the type intact when mapping from a formats that have > strongly typed options (example: Protobuf). > This is part of a multi ticket implementation. The following tickets are > related: > # Typed options for Row Schema and Fields > # Convert Proto Options to Beam Schema options > # Convert Avro extra information for Beam string options > # Replace meta data with Logical Type options > # Extract meta data in Calcite SQL to Beam options > # Extract meta data in Zeta SQL to Beam options > # Add java example of using option in a transform > This feature is discussed with Reuven Lax, Brian Hulette -- This message was sent by Atlassian Jira (v8.3.4#803005)