Re: k8s job cluster using StatefulSet

2020-08-16 Thread Yang Wang
ank you Arvid and Yang! > > -- > *From:* Yang Wang > *Sent:* Thursday, August 13, 2020 8:09:13 PM > *To:* Arvid Heise > *Cc:* Alexey Trenikhun ; user > *Subject:* Re: k8s job cluster using StatefulSet > > Hi Alexey, > > Actually, Stateful

Re: k8s job cluster using StatefulSet

2020-08-14 Thread Alexey Trenikhun
Thank you Arvid and Yang! From: Yang Wang Sent: Thursday, August 13, 2020 8:09:13 PM To: Arvid Heise Cc: Alexey Trenikhun ; user Subject: Re: k8s job cluster using StatefulSet Hi Alexey, Actually, StatefulSets could also be used to start the JobManager and

Re: k8s job cluster using StatefulSet

2020-08-14 Thread Jan Lukavský
Hi Alexey, I'm using StatefulSet for JM exactly as you describe (Deployment for TM is just fine). The main advantage is that you don't need distributed storage for JM fault tolerance, because you can use persistent volume mount (provided your cloud provider provides it as fault tolerant volum

Re: k8s job cluster using StatefulSet

2020-08-13 Thread Yang Wang
Hi Alexey, Actually, StatefulSets could also be used to start the JobManager and TaskManager. So why do we suggest to use Deployment in the Flink documentation? * StatefulSets requires the user to have persistent volume in the K8s cluster. However, it is not always true, especially for the unma

Re: k8s job cluster using StatefulSet

2020-08-13 Thread Arvid Heise
Hi Alexey, I don't see any issue in using stateful sets immediately. I'd recommend using one of the K8s operators or Ververica's community edition [1] though if you start with a new setup as they may solve even more issues that you might experience in the future. [1] https://www.ververica.com/ge

k8s job cluster using StatefulSet

2020-08-10 Thread Alexey Trenikhun
Hello, Flink documentation suggests to use Deployments to deploy JM and TM for kubernetes job cluster. Is any known potential issues with using StatefulSets instead, seems StatefullSet provides uniqueness for JM during upgrade/rollback, while with Deployments could be multiple JM pods (e.g.1 ter