Thank you very much for the option provided. Just checked the stackoverflow
settings and tried at our end. It can be an option for us.
Thank you.
Best,
*Xiao Ma*
On Tue, Oct 11, 2022 at 7:41 PM Lydian <lydia...@gmail.com> wrote:
> You should be able to provide extra_args to the expansion service:
> ```
>
> --defaultEnvironmentType=PROCESS
> --defaultEnvironmentConfig={"command":"/opt/apache/beam_java/boot"}
>
> ```
>
> I am also running the Xlang pipeline in flink k8s cluster. After setting
> the defaultEnvironmentType to PROCESS, I don't need to use DinD or DooD at
> all.
> You can also find my full settings in:
> https://stackoverflow.com/a/74035035/19259600
>
>
>
> Sincerely,
> Lydian Lee
>
>
>
> On Tue, Oct 11, 2022 at 11:34 AM Xiao Ma <xia...@geotab.com> wrote:
>
>> The worker pool means `starting a java or python sdk`, to accept the java
>> or python pipeline running. For example, to execute python pipeline, we
>> have to start a python worker pool with `--worker_pool` arguments. For the
>> Java code, besides the docker mode (default one), do we have other better
>> ways to start a java worker pool?
>>
>> For now, our flink cluster is running on the k8s. If we choose the
>> default sdk harness mode (docker), we will have the docker (java sdk
>> harness) in docker (flink-taskmanager). So, what we are doing is to call
>> org.apache.beam.fn.harness.ExternalWorkerService class with pipeline
>> options as environment variables and fixed two small issues in the
>> FnHarness class to make sure the java sdk harness can run smoothly.
>>
>> Thank you.
>> *Mark Ma*
>>
>>
>> On Tue, Oct 11, 2022 at 12:46 PM Alexey Romanenko <
>> aromanenko....@gmail.com> wrote:
>>
>>> I’m not sure that I get it correctly. What do you mean by “worker pool”
>>> in your case?
>>>
>>> —
>>> Alexey
>>>
>>> On 8 Oct 2022, at 03:24, Xiao Ma <xia...@geotab.com> wrote:
>>>
>>> Hello,
>>>
>>> I would like to run a pipeline with Java as the main language and python
>>> transformation embedded. The beam pipeline is running on the flink cluster.
>>> Currently, I can run it with a taskmanager + java worker pool and a python
>>> worker pool. Could I ask if there is a way to run the java code on the task
>>> manager directly and keep the python worker pool?
>>>
>>> Current: taskmanager + java worker pool + python worker pool
>>> Desired: taskmanager + python worker pool
>>>
>>> Thank you very much.
>>>
>>> *Mark Ma*
>>>
>>>
>>>
