Hi Beamers,
We can use artifact staging to make sure SDK workers have access to a
pipeline's dependencies. However, artifact staging is not always necessary.
For example, one can make sure that the environment contains all the
dependencies ahead of time. However, regardless of whether or not artif
Certainly there's a lot to be re-thought in terms of artifact staging,
especially when it comes to cross-langauge pipelines. I think it would
makes sense to have a special retrieval token for the "empty"
manifest, which would mean a staging directory would never have to be
set up if no artifacts ha
FWIW, there are also discussions of adding a preparation phase for sdk
harness (docker) images, such that artifacts could be staged (and
installed, compiled etc.) ahead of time and shipped as part of the sdk
image rather than via a side channel (and on every worker). Anyone not
using these images i
I'm running into the issue Kyle points out when I try to run a pipeline
that does not use artifact staging:
2019-11-23 01:09:18,442 WARN
org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService
- GetManifest for
/tmp/beam-artifact-staging/job_53cad419-a8c0-472c-8486-f795cc88
JIRA: https://issues.apache.org/jira/browse/BEAM-8815
On Fri, Nov 22, 2019 at 5:31 PM Thomas Weise wrote:
> I'm running into the issue Kyle points out when I try to run a pipeline
> that does not use artifact staging:
>
> 2019-11-23 01:09:18,442 WARN
> org.apache.beam.runners.fnexecution.artif
boot.go could be updated to recognize NO_ARTIFACTS_STAGED_TOKEN as
well. (Should this constant be put in a common location?)
On Sat, Nov 23, 2019 at 9:16 AM Thomas Weise wrote:
>
> JIRA: https://issues.apache.org/jira/browse/BEAM-8815
>
>
> On Fri, Nov 22, 2019 at 5:31 PM Thomas Weise wrote:
>>
When we added the class loader artifact stager, we introduced artifact
retrieval service type as a pipeline option. It would make sense to put a
"none" option there.
https://github.com/apache/beam/blob/5fd93af49e6cb86ff52b20f103371df7e0447b7f/sdks/java/core/src/main/java/org/apache/beam/sdk/option
Thanks, I would prefer to solve this in a way where the user does not need
to configure anything extra though.
On Mon, Nov 25, 2019 at 10:21 AM Kyle Weaver wrote:
> When we added the class loader artifact stager, we introduced artifact
> retrieval service type as a pipeline option. It would mak
Ah didn't see your pull request yet Thomas. Will take a look later.
On Mon, Nov 25, 2019 at 10:23 AM Thomas Weise wrote:
> Thanks, I would prefer to solve this in a way where the user does not need
> to configure anything extra though.
>
>
> On Mon, Nov 25, 2019 at 10:21 AM Kyle Weaver wrote:
>