Thanks Cham! I really like the proposal, I left a few comments. I also had one higher-level point I wanted to elevate here:
> Pipeline SDKs can generate user-friendly stub-APIs based on transforms registered with an expansion service, eliminating the need to develop language-specific wrappers. This would be great! I think one point to consider is whether we can do this statically. We could package up these stubs with releases and include them in API docs for each language, making them much more discoverable. That could be an extension on top of your proposal (e.g. as part of its build, each SDK spins up other known expansion services and generates code based on the discovery responses), but maybe it could be cleaner if we don't really need the dynamic version? Brian On Thu, Aug 4, 2022 at 6:51 PM Chamikara Jayalath via dev < dev@beam.apache.org> wrote: > Hi All, > > I believe we can make the multi-language pipelines offering [1] much > easier to use by updating the expansion service to be fully aware of > SchemaTransforms. Additionally this will make it easy to > register/discover/use transforms defined in one SDK from all other SDKs. > Specifically we could add the following features. > > - Expansion service can be used to easily initialize and expand > transforms without need for additional code. > - Expansion service can be used to easily discover already registered > transforms. > - Pipeline SDKs can generate user-friendly stub-APIs based on > transforms registered with an expansion service, eliminating the need to > develop language-specific wrappers. > > Please see here for my proposal: https://s.apache.org/easy-multi-language > > Lemme know if you have any comments/questions/suggestions :) > > Thanks, > Cham > > [1] > https://beam.apache.org/documentation/programming-guide/#multi-language-pipelines > >
