I like the capabilities/requirements idea. Would these capabilities be at a level that it would make sense to document in the capabilities matrix? i.e. could the URNs be the values of "X" Pablo described here [1].
Brian [1] https://lists.apache.org/thread.html/e93ac64d484551d61e559e1ba0cf4a15b760e69d74c5b1d0549ff74f%40%3Cdev.beam.apache.org%3E On Mon, Feb 10, 2020 at 3:55 PM Robert Bradshaw <rober...@google.com> wrote: > With an eye towards cross-language (which includes cross-version) > pipelines and services (specifically looking at Dataflow) supporting > portable pipelines, there's been a desire to stabilize the portability > protos. There are currently many cleanups we'd like to do [1] (some > essential, others nice to have); are there others that people would > like to see? > > Of course we would like it to be possible for the FnAPI and Beam > itself to continue to evolve. Most of this can be handled by runners > understanding various transform URNs, but not all. (An example that > comes to mind is support for large iterables [2], or the requirement > to observe and respect new fields on a PTransform or its payloads > [3]). One proposal for this is to add capabilities and/or > requirements. An environment (corresponding generally to an SDK) could > adveritize various capabilities (as a list or map of URNs) which a > runner can take advantage of without requiring all SDKs to support all > features at the same time. For the other way around, we need a way of > marking something that a runner must reject if it does not understand > it. This could be a set of requirements (again, a list of map of URNs) > that designate capabilities required to at least be understood by the > runner to faithfully execute this pipeline. (These could be attached > to a transform or the pipeline itself.) Do these sound like reasonable > additions? Also, would they ever need to be parameterized (map), or > would a list suffice? > > [1] BEAM-2645, BEAM-2822, BEAM-3203, BEAM-3221, BEAM-3223, BEAM-3227, > BEAM-3576, BEAM-3577, BEAM-3595, BEAM-4150, BEAM-4180, BEAM-4374, > BEAM-5391, BEAM-5649, BEAM-8172, BEAM-8201, BEAM-8271, BEAM-8373, > BEAM-8539, BEAM-8804, BEAM-9229, BEAM-9262, BEAM-9266, and BEAM-9272 > [2] > https://lists.apache.org/thread.html/70cac361b659516933c505b513d43986c25c13da59eabfd28457f1f2@%3Cdev.beam.apache.org%3E > [3] > https://lists.apache.org/thread.html/rdc57f240069c0807eae87ed2ff13d3ee503bc18e5f906d05624e6433%40%3Cdev.beam.apache.org%3E >
