External environments are mainly used for testing since they represent environments that are already running. There are other clever uses of it as well but unlikely what your looking for.
The docker way will be nice once there is integration between the FlinkRunner and its supported cluster managers to automatically manage the SDK container instances. Until that exists you could look at using the process environment (it launches the SDK as a child process beside the Flink workers). On Tue, Sep 10, 2019 at 4:38 AM Andrea Medeghini < andrea.medegh...@10xbanking.com> wrote: > Hello, > > Can I run Beam Python pipelines with Flink deployed on Kubernetes? > > > > I have successfully managed to run a Beam Python pipeline using the > Portable runner and the job service pointing to a local Flink server > running as Docker containers. > > I was able to achieve that mounting the Docker socket in my Flink > container and running Flink as root, so the class DockerEnvironmentFactory > can create the Python harness container. > > Unfortunately, I can't use the same solution when Flink is running in > Kubernetes. Moreover, I don't want to create the Python harness container > using the Docker command from my pods. > > I noticed there is an implementation called ExternalEnvironmentFactory, > but I am not sure how to use it. > > Is there a way to deploy a side container and use a different factory to > run the Python harness process? > > What is the correct approach? > > > > Thanks, > > Andrea > > > > > > > [image: 10x] <https://www.10xbanking.com/> > > 10x Banking > > Please consider the environment before printing this e-mail > > ********************************************************************************************************************************************************** > This e-mail, including any attachments, is confidential. If you are not > the intended recipient, you may not copy, forward, disclose or use any part > of it. If you have received this e-mail in error, please notify the sender > immediately by return e-mail and delete this message and any copies and > attachments. > > 10x Future Technologies Services Limited is registered in England & Wales > Registered Number: 11262659 > Registered Address: 5 Howick Place, Victoria, London, SW1P 1WG >
