On Fri, Oct 7, 2022 at 6:29 PM Xiao Ma <[email protected]> wrote: > Hello, > > I would like to run a pipeline with Java as the main language and python > transformation embedded. The beam pipeline is running on the flink cluster. > Currently, I can run it with a taskmanager + java worker pool and a python > worker pool. Could I ask if there is a way to run the java code on the task > manager directly and keep the python worker pool? > > Current: taskmanager + java worker pool + python worker pool > Desired: taskmanager + python worker pool >
Generally this is not possible. If the transform has to be executed on the SDK side, the runner usually sets up an environment (for example, a Docker container) with the corresponding SDK and executes the bundles with the transform using the Beam Fn API. Runners can choose to override this by executing the transform within the runner itself, but you'll have to modify the Flink runner to do this. Thanks, Cham > > Thank you very much. > > *Mark Ma* > >
