Hello All,
We have a use case where in a java portable pipeline, we would like to have
multiple environments setup in order that some executable stage runs in one
environment while some other executable stages runs in another environment.
Couple of questions on this:
1. Is this current support
Thanks Luke for the reply, do you know what is the preferred way to configure a
PTransform to be executed in a different environment from another PTransform
when both are in the same SDK, e.g. Java ?
Best,
Ke
> On Sep 21, 2021, at 9:48 PM, Luke Cwik wrote:
>
> Environments that aren't exactly
Thanks for the advice.
Here are some more background:
We are building a feature called “split deployment” such that, we can isolate
framework/platform core from user code/dependencies to address couple of
operational challenges such as dependency conflict, alert/exception triaging.
With Beam’s
Ideally, we do not want to expose anything directly to users and we, as the
framework and platform provider, separate things out under the hood.
I would expect users to author their DoFn(s) in the same way as they do right
now, but we expect to change the DoFn(s) that we provide, will be
annot
Go SDK Note: At present there's no implementation for using Go SDK
transforms in Python or Java, and due to Go's Static Compilation approach,
it's not set in stone for how that will happen.
Eg. Tiny self contained binaries? A plug-in framework? Neither would run
into the same runtime dependency co
I agree! But the Go plugin model might permit fusion between different
plugins avoiding unnecessary serialization between multiple Go xlang
DoFns While still avoiding the Diamond Dep problem, since the plugins
are binaries themselves.
Not sure for certain as I haven't played around with it yet
On Thu, Sep 30, 2021 at 9:25 AM Ke Wu wrote:
>
> Ideally, we do not want to expose anything directly to users and we, as the
> framework and platform provider, separate things out under the hood.
>
> I would expect users to author their DoFn(s) in the same way as they do right
> now, but we expe
I am able to annotate/mark a java transform by setting its resource hints [1]
as well, which resulted in a different environment id, e.g.
beam:env:docker:v1 VS beam:env:docker:v11
Is this on the right track? If Yes, I suppose then I need to configure job
bundle factory to be able to understan
This is great, let me try it out.
Best,
Ke
> On Sep 30, 2021, at 6:06 PM, Robert Bradshaw wrote:
>
> On Thu, Sep 30, 2021 at 6:00 PM Ke Wu wrote:
>>
>> I am able to annotate/mark a java transform by setting its resource hints
>> [1] as well, which resulted in a different environment id, e.g.
I have a quick follow up questions.
When using multiple external environments, is there a way to configure the
multiple external service address? It looks like the current PipelineOptions
only supports specifying one external address.
Best,
Ke
> On Oct 4, 2021, at 4:12 PM, Ke Wu wrote:
>
>
Hi Robert,
Thanks for the pointer, using expansion service hack seems to work!
On the other hand, since PipelineOptions is the place to configure external
service address anyway, do you think it makes sense to expand it so it is
capable of specifying multiple external environment to external se
Hi Robert,
After some more digging and exploration, I realized that it is not clean and
straightforward to update PipelineOption to support such.
Therefore, I spent more time exploring the external transform & expansion
service route. However, I noticed that this approach has the same limitati
Awesome, I am happy to make the update.
One more question, I noticed that expansion service is currently being brought
up as part of job server in Java [1], if this is the preferred approach, does
it mean, we should update ServerConfiguration to include such configurations,
like expansion serve
Agreed, that is why I was hesitant to complicate ServerConfiguration such that
it can configure the expansion service it brings up.
In this case, do you think we may just add one more option to disable the
expansion service setup so for complicated use cases, expansion service can be
brought up
Thanks for Cham to chime in, having each expansion service instance to be able
to serve a distinct environment which is configurable is what I am looking for,
however, I don’t think it is practical atm.
The pipeline option it uses to register environment is NOT the pipeline option
of expansion
Thanks for the pointer! I missed the part that pipeline’s options is copied
from expansion options!
Then, I think I just need a way to disable the auto creation of expansion
service during job server start up.
> On Dec 15, 2021, at 11:29 AM, Chamikara Jayalath wrote:
>
>
>
> On Wed, Dec 1
16 matches
Mail list logo
