Hello beam community!
I’m looking for some help solving an issue running a beam job on flink
using --environment_type DOCKER.
I have a flink cluster running in kubernetes configured so the taskworkers
can run docker [1]. I’m attempting to deploy a flink jobserver in the
cluster. The issue is that
> The issue is that the jobserver does not provide the proper endpoints to
the SDK harness when it submits the job to flink.
I would not be surprised if there was something weird going on with Docker
in Docker. The defaults mostly work fine when an external SDK harness is
used [1].
Can you provid
I would not be surprised if there was something weird going on with Docker
> in Docker. The defaults mostly work fine when an external SDK harness is
> used [1].
>
Can you provide more information on the exception you got? (I'm
> particularly interested in the line number).
>
The actual error is a
It was my understanding that the client first uploads the artifacts to the
jobserver and then the SDK harness will pull in these artifacts from the
jobserver over a gRPC port.
I see the artifacts on the jobserver while the job is attempting to run:
root@flink-beam-jobserver-9fccb99b8-6mhtq
:/tmp/
> It was my understanding that the client first uploads the artifacts to
the jobserver and then the SDK harness will pull in these artifacts from
the jobserver over a gRPC port.
Not quite. The artifact endpoint is for communicating which artifacts are
needed, and where to find them. But the SDK har
