Go SDK Note: At present there's no implementation for using Go SDK transforms in Python or Java, and due to Go's Static Compilation approach, it's not set in stone for how that will happen.
Eg. Tiny self contained binaries? A plug-in framework? Neither would run into the same runtime dependency collision java and python have on occasion (they'd be resolved at compile time anyway...) On Thu, Sep 30, 2021, 9:25 AM Ke Wu <ke.wu...@gmail.com> wrote: > Ideally, we do not want to expose anything directly to users and we, as > the framework and platform provider, separate things out under the hood. > > I would expect users to author their DoFn(s) in the same way as they do > right now, but we expect to change the DoFn(s) that we provide, will be > annotated/marked so that it can be recognized during runtime. > > In our use case, application is executed in Kubernetes environment > therefore, we are expecting to directly use different docker image to > isolate dependencies. > > e.g. we have docker image A, which is beam core, that is used to start job > server and runner process. We have a docker image B, which contains DoFn(s) > that platform provides to serve as a external worker pool service to > execute platform provided DoFn(s), last but not least, users would have > their own docker image represent their application, which will be used to > start the external worker pool service to handle their own UDF execution. > > Does this make sense ? > > On Sep 29, 2021, at 1:09 PM, Luke Cwik <lc...@google.com> wrote: > > That sounds neat. I think that before you try to figure out how to change > Beam to fit this usecase is to think about what would be the best way for > users to specify these requirements when they are constructing the > pipeline. Once you have some samples that you could share the community > would probably be able to give you more pointed advice. > For example will they be running one application with a complicated class > loader setup, if so then we could probably do away with multiple > environments and try to have DoFn's recognize their specific class loader > configuration and replicate it on the SDK harness side. > > Also, for performance reasons users may want to resolve their dependency > issues to create a maximally fused graph to limit performance impact due to > the encoding/decoding boundaries at the edges of those fused graphs. > > Finally, this could definitely apply to languages like Python and Go (now > that Go has support for modules) as dependency issues are a common problem. > > > On Wed, Sep 29, 2021 at 11:47 AM Ke Wu <ke.wu...@gmail.com> wrote: > >> Thanks for the advice. >> >> Here are some more background: >> >> We are building a feature called “split deployment” such that, we can >> isolate framework/platform core from user code/dependencies to address >> couple of operational challenges such as dependency conflict, >> alert/exception triaging. >> >> With Beam’s portability framework, runner and sdk worker process >> naturally decouples beam core and user UDFs(DoFn), which is awesome! On top >> of this, we could further distinguish DoFn(s) that end user authors from >> DoFn(s) that platform provides, therefore, we would like these DoFn(s) to >> be executed in different environments, even in the same language, e.g. Java. >> >> Therefore, I am exploring approaches and recommendations what are the >> proper way to do that. >> >> Let me know your thoughts, any feedback/advice is welcome. >> >> Best, >> Ke >> >> On Sep 27, 2021, at 11:56 AM, Luke Cwik <lc...@google.com> wrote: >> >> Resource hints have a limited use case and might fit your need. >> You could also try to use the expansion service XLang route to bring in a >> different Java environment. >> Finally, you could modify the pipeline proto that is generated directly >> to choose which environment is used for which PTransform. >> >> Can you provide additional details as to why you would want to have two >> separate java environments (e.g. incompatible versions of libraries)? >> >> On Wed, Sep 22, 2021 at 3:41 PM Ke Wu <ke.wu...@gmail.com> wrote: >> >>> Thanks Luke for the reply, do you know what is the preferred way to >>> configure a PTransform to be executed in a different environment from >>> another PTransform when both are in the same SDK, e.g. Java ? >>> >>> Best, >>> Ke >>> >>> On Sep 21, 2021, at 9:48 PM, Luke Cwik <lc...@google.com> wrote: >>> >>> Environments that aren't exactly the same are already in separate >>> ExecutableStages. The GreedyPCollectionFuser ensures that today[1]. >>> >>> Workarounds like getOnlyEnvironmentId would need to be removed. It may >>> also be effectively dead-code. >>> >>> 1: >>> https://github.com/apache/beam/blob/ebf2aacf37b97fc85b167271f184f61f5b06ddc3/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java#L144 >>> >>> On Tue, Sep 21, 2021 at 1:45 PM Ke Wu <ke.wu...@gmail.com> wrote: >>> >>>> Hello All, >>>> >>>> We have a use case where in a java portable pipeline, we would like to >>>> have multiple environments setup in order that some executable stage runs >>>> in one environment while some other executable stages runs in another >>>> environment. Couple of questions on this: >>>> >>>> 1. Is this current supported? I noticed a TODO in [1] which suggests it >>>> is feature pending support >>>> 2. If we did support it, what would the ideal mechanism to distinguish >>>> ParDo/ExecutableStage to be executed in different environment, is it >>>> through ResourceHints? >>>> >>>> >>>> Best, >>>> Ke >>>> >>>> >>>> [1] >>>> https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java#L344 >>>> >>>> >>> >>> >> >
