Seems like a useful thing to me and will make it easier for Beam users overall.
On Fri, Feb 10, 2023 at 3:56 PM Robert Bradshaw via dev <dev@beam.apache.org> wrote: > Thanks. I added some comments to the doc. > > On Mon, Feb 6, 2023 at 1:33 PM Chamikara Jayalath via dev > <dev@beam.apache.org> wrote: > > > > Hi All, > > > > Beam PTransforms are currently primarily identified as operations in a > pipeline that perform specific tasks. PTransform implementations were > traditionally linked to specific Beam SDKs. > > > > With the advent of portability framework, multi-language pipelines, and > expansion services that can be used to build/expand and discover > transforms, we have an opportunity to make this more general and > re-introduce Beam PTransforms as computation units that can serve any > use-case that needs to discover or use Beam transforms. For example, any > Beam SDK that runs a pipeline using a portable Beam runner should be able > to use a transform offered through an expansion service irrespective of the > implementation SDK of the transform or the pipeline. > > > > I believe we can make such use-cases much easier to manage by > introducing a user-deployable service that encapsulates existing Beam > expansion services in the form of a Kubernetes cluster. The service will > offer a single gRPC endpoint and will include Beam expansion services > developed in different languages. Any Beam pipeline, irrespective of the > pipeline SDK, should be able to use any transform offered by the service. > > > > This will also offer a way to make multi-language pipeline execution, > which currently relies on locally downloaded large dependencies and locally > started expansion service processes, more robust. > > > > I have written a proposal for implementing such a service and it's > available at https://s.apache.org/beam-transform-service. > > > > Please take a look and let me know if you have any comments or questions. > > > > Thanks, > > Cham >
