Using schema types for the metadata values is a nice touch. Are the options expected to be common across many fields? Perhaps the name should be a URN to make it clear to be careful about collisions? (just a synonym for "name" in practice, but with different connotation)
I generally like this... but the examples (all but one) are weird things that I don't really understand how they are done or who is responsible for them. One way to go is this: if options are maybe not understood by all consumers, then they can't really change behavior. Kind of like how URN and payload on a composite transform can be ignored and just the expansion used. Kenn On Sun, Jan 26, 2020 at 8:27 AM Alex Van Boxel <a...@vanboxel.be> wrote: > Hi everyone, > > I'm proud to announce my first real proposal. The proposal describes Beam > Schema Options. This is an extension to the Schema API to add typed meta > data to to Rows, Field and Logical Types: > > > https://docs.google.com/document/d/1yCCRU5pViVQIO8-YAb66VRh3I-kl0F7bMez616tgM8Q/edit?usp=sharing > > To give you some context where this proposal comes from: We've been using > dynamic meta driven pipelines for a while, but till now in an awkward and > hacky way (see my talks at the previous Beam Summits). This proposal would > bring a good way to work with metadata on the metadata :-). > > The proposal points to 2 pull requests with the implementation, one for > the API another for translating proto options to beam options. > > Thanks to Brian Hulette and Reuven Lax for the initial feedback. All > feedback is welcome. > > _/ > _/ Alex Van Boxel >
