Adding a few relevant folks who could help answer this question: @John Casey <[email protected]> @Chamikara Jayalath <[email protected]> @Robert Bradshaw <[email protected]>
Lydian, if you have any other information please share an update. Ahmet On Tue, Jun 7, 2022 at 12:49 AM Lydian <[email protected]> wrote: > Hi Folks, > > I am trying to set up the Beam environment to run our Python pipeline > which reads data from Kafka. According to some previous thread > <https://lists.apache.org/thread/kz47y88t6zr9k4z043mx3wnb9mz5dqpq>, it > seems like the Java SDK harness doesn't work with PROCESS environment_type, > and thus I can only use either Docker or External. Given that I need to > deploy the job to K8s, and there are some security concerns which prevent > me from using the DinD approach. It seems like my best solution is to start > a sidecar container which starts the Java extension service in Flink Task > manager. However, I am not sure what's the command to start the java > extension service which supports the approach. > > It looks like in the Docker environment, it is trying to call > ``` > /opt/apache/beam/boot --id=1-2 --provision_endpoint=localhost:33025 > ``` > But the script requires me to provide `id` and `provision_endpoint`, which > I am not sure what I should do if I want to set up an external service. > Wondering if someone can help me with this? > > for context, I am using beam 2.38.0 with flink 1.13, the job is deployed > to k8s using lyft/flinkk8soperator > <https://github.com/lyft/flinkk8soperator> > > Thanks! >
