Hi, I'm looking for advice on how to run flink streaming jobs on Yarn cluster in production environment. I tried in testing environment both approaches with HA mode, namely yarn session + multiple jobs vs cluster per job, both seems to work for my cases, with slight preference of yarn session mode to centrally manage credentials. I'm looking to run about 10 streaming jobs mostly reading/writing from kafka + cassandra with following restictions: 1. yarn nodes will be hard rebooted quite often, roughly every 2 weeks. I have a concern here what happens when Job manager dies in session mode. 2. there are often network interruptions/slowdowns. 3. I'm trying to minimise time to restart job to have as much as possible continious processing.
Thanks in advance, Maxim.
