Running Beam on a native Kubernetes Flink cluster

2021-08-14 Thread Gorjan Todorovski
Hi! I need help implementing a native Kubernetes Flink cluster that needs to run batch jobs (run by TensorFlow Extended), but I am not sure I am configuring it right as I have issues running jobs on more than one task manager, while jobs run fine if there is only one TM. I use the following param

Re: Running Beam on a native Kubernetes Flink cluster

2021-08-15 Thread Jan Lukavský
Hi Gorjan, the address of localhost is hard-coded in the python worker pool (see [1]). There should be no need to setup a load-balancer for the worker_pool, if you have it as another container in each TM pod, it should suffice to replace {beam_sdk_url} with 'localhost'. Each TM will then have

Re: Running Beam on a native Kubernetes Flink cluster

2021-08-15 Thread Gorjan Todorovski
Yes! That did it. Changed to localhost and all works fine now. I was wrong thinking it would like to connect to Beam SDK worker from my client machine, hence i added the load balancer. Thank you Jan! On Sun, 15 Aug 2021 at 16:45, Jan Lukavský wrote: > Hi Gorjan, > > the address of localhost is