There have been a few comments in the doc and I'm going to start working on the worker execution part.
Instead of running a Docker container for each worker, the preferred option is to use the worker pool: https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.bzs6y6ms0898 There are some notes at the end of the doc regarding the implementation. It would be great if those interested in the environments and Python docker container could take a look. In a nutshell, the proposal is to make few changes to the Python container so that it (optionally) can be used to run the worker pool. Thanks, Thomas On Wed, Jul 24, 2019 at 9:42 PM Pablo Estrada <[email protected]> wrote: > I am very happy to see this. I'll take a look, and leave my comments. > > I think this is something we'd been needing, and it's great that you guys > are putting thought into it. Thanks!<3 > > On Wed, Jul 24, 2019 at 9:01 PM Thomas Weise <[email protected]> wrote: > >> Hi, >> >> Recently Lyft open sourced *FlinkK8sOperator,* a Kubernetes operator to >> manage Flink deployments on Kubernetes: >> >> https://github.com/lyft/flinkk8soperator/ >> >> We are now discussing how to extend this operator to also support >> deployment of Python Beam pipelines with the Flink runner. I would like to >> share the proposal with the Beam community to enlist feedback as well as >> explore opportunities for collaboration: >> >> >> https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/ >> >> Looking forward to your comments and suggestions! >> >> Thomas >> >>
