Thanks Luke for the reply, do you know what is the preferred way to configure a PTransform to be executed in a different environment from another PTransform when both are in the same SDK, e.g. Java ?
Best, Ke > On Sep 21, 2021, at 9:48 PM, Luke Cwik <[email protected]> wrote: > > Environments that aren't exactly the same are already in separate > ExecutableStages. The GreedyPCollectionFuser ensures that today[1]. > > Workarounds like getOnlyEnvironmentId would need to be removed. It may also > be effectively dead-code. > > 1: > https://github.com/apache/beam/blob/ebf2aacf37b97fc85b167271f184f61f5b06ddc3/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java#L144 > > <https://github.com/apache/beam/blob/ebf2aacf37b97fc85b167271f184f61f5b06ddc3/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java#L144> > On Tue, Sep 21, 2021 at 1:45 PM Ke Wu <[email protected] > <mailto:[email protected]>> wrote: > Hello All, > > We have a use case where in a java portable pipeline, we would like to have > multiple environments setup in order that some executable stage runs in one > environment while some other executable stages runs in another environment. > Couple of questions on this: > > 1. Is this current supported? I noticed a TODO in [1] which suggests it is > feature pending support > 2. If we did support it, what would the ideal mechanism to distinguish > ParDo/ExecutableStage to be executed in different environment, is it through > ResourceHints? > > > Best, > Ke > > > [1] > https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java#L344 > > <https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java#L344> >
