The artifact staging directory (configurable via the "--artifacts_dir"
pipeline option) needs to be accessible to workers, which in this case are
containerized. There are a couple workarounds:
1. Don't stage files through the file system at all, i.e. use
--runner=FlinkRunner in Python instead of -
Thanks for the details in the gist. I would guess the key to the error is
this:
2021/05/24 13:45:34 Initializing java harness: /opt/apache/beam/boot
--id=1-2 --provision_endpoint=localhost:33957
2021/05/24 13:45:44 Failed to retrieve staged files: failed to retrieve
/tmp/staged in 3 attemp
