Thank you very much for the option provided. Just checked the stackoverflow settings and tried at our end. It can be an option for us.
Thank you. Best, *Xiao Ma* On Tue, Oct 11, 2022 at 7:41 PM Lydian <lydia...@gmail.com> wrote: > You should be able to provide extra_args to the expansion service: > ``` > > --defaultEnvironmentType=PROCESS > --defaultEnvironmentConfig={"command":"/opt/apache/beam_java/boot"} > > ``` > > I am also running the Xlang pipeline in flink k8s cluster. After setting > the defaultEnvironmentType to PROCESS, I don't need to use DinD or DooD at > all. > You can also find my full settings in: > https://stackoverflow.com/a/74035035/19259600 > > > > Sincerely, > Lydian Lee > > > > On Tue, Oct 11, 2022 at 11:34 AM Xiao Ma <xia...@geotab.com> wrote: > >> The worker pool means `starting a java or python sdk`, to accept the java >> or python pipeline running. For example, to execute python pipeline, we >> have to start a python worker pool with `--worker_pool` arguments. For the >> Java code, besides the docker mode (default one), do we have other better >> ways to start a java worker pool? >> >> For now, our flink cluster is running on the k8s. If we choose the >> default sdk harness mode (docker), we will have the docker (java sdk >> harness) in docker (flink-taskmanager). So, what we are doing is to call >> org.apache.beam.fn.harness.ExternalWorkerService class with pipeline >> options as environment variables and fixed two small issues in the >> FnHarness class to make sure the java sdk harness can run smoothly. >> >> Thank you. >> *Mark Ma* >> >> >> On Tue, Oct 11, 2022 at 12:46 PM Alexey Romanenko < >> aromanenko....@gmail.com> wrote: >> >>> I’m not sure that I get it correctly. What do you mean by “worker pool” >>> in your case? >>> >>> — >>> Alexey >>> >>> On 8 Oct 2022, at 03:24, Xiao Ma <xia...@geotab.com> wrote: >>> >>> Hello, >>> >>> I would like to run a pipeline with Java as the main language and python >>> transformation embedded. The beam pipeline is running on the flink cluster. >>> Currently, I can run it with a taskmanager + java worker pool and a python >>> worker pool. Could I ask if there is a way to run the java code on the task >>> manager directly and keep the python worker pool? >>> >>> Current: taskmanager + java worker pool + python worker pool >>> Desired: taskmanager + python worker pool >>> >>> Thank you very much. >>> >>> *Mark Ma* >>> >>> >>>