Thanks for the advice. Here are some more background:
We are building a feature called “split deployment” such that, we can isolate framework/platform core from user code/dependencies to address couple of operational challenges such as dependency conflict, alert/exception triaging. With Beam’s portability framework, runner and sdk worker process naturally decouples beam core and user UDFs(DoFn), which is awesome! On top of this, we could further distinguish DoFn(s) that end user authors from DoFn(s) that platform provides, therefore, we would like these DoFn(s) to be executed in different environments, even in the same language, e.g. Java. Therefore, I am exploring approaches and recommendations what are the proper way to do that. Let me know your thoughts, any feedback/advice is welcome. Best, Ke > On Sep 27, 2021, at 11:56 AM, Luke Cwik <lc...@google.com> wrote: > > Resource hints have a limited use case and might fit your need. > You could also try to use the expansion service XLang route to bring in a > different Java environment. > Finally, you could modify the pipeline proto that is generated directly to > choose which environment is used for which PTransform. > > Can you provide additional details as to why you would want to have two > separate java environments (e.g. incompatible versions of libraries)? > > On Wed, Sep 22, 2021 at 3:41 PM Ke Wu <ke.wu...@gmail.com > <mailto:ke.wu...@gmail.com>> wrote: > Thanks Luke for the reply, do you know what is the preferred way to configure > a PTransform to be executed in a different environment from another > PTransform when both are in the same SDK, e.g. Java ? > > Best, > Ke > >> On Sep 21, 2021, at 9:48 PM, Luke Cwik <lc...@google.com >> <mailto:lc...@google.com>> wrote: >> >> Environments that aren't exactly the same are already in separate >> ExecutableStages. The GreedyPCollectionFuser ensures that today[1]. >> >> Workarounds like getOnlyEnvironmentId would need to be removed. It may also >> be effectively dead-code. >> >> 1: >> https://github.com/apache/beam/blob/ebf2aacf37b97fc85b167271f184f61f5b06ddc3/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java#L144 >> >> <https://github.com/apache/beam/blob/ebf2aacf37b97fc85b167271f184f61f5b06ddc3/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java#L144> >> On Tue, Sep 21, 2021 at 1:45 PM Ke Wu <ke.wu...@gmail.com >> <mailto:ke.wu...@gmail.com>> wrote: >> Hello All, >> >> We have a use case where in a java portable pipeline, we would like to have >> multiple environments setup in order that some executable stage runs in one >> environment while some other executable stages runs in another environment. >> Couple of questions on this: >> >> 1. Is this current supported? I noticed a TODO in [1] which suggests it is >> feature pending support >> 2. If we did support it, what would the ideal mechanism to distinguish >> ParDo/ExecutableStage to be executed in different environment, is it through >> ResourceHints? >> >> >> Best, >> Ke >> >> >> [1] >> https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java#L344 >> >> <https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java#L344> >> >
