I'm about to make the switch to Kubernetes with Airflow, but am wondering what happens when my CI/CD pipeline redeploys the webserver and scheduler and there are still long-running tasks (pods). My intuition is that since the database hold all state and the tasks are in charge of updating their own state, and the UI only renders what it sees in the database that this is not so much of a problem. To be sure, however, here are my questions:
Will task pods continue to run? Can task pods continue to poll the external system they are running tasks on while being "headless"? Can the tasks pods change/update state in the database while being "headless"? Will the UI/Scheduler still be aware of the tasks (pods) once they are live again? Is there anything else the might cause issues when deploying while tasks (pods) are running that I'm not thinking of here? Kyle Hamlin
