sdaberdaku opened a new pull request, #76749: URL: https://github.com/apache/airflow/pull/76749
Kubernetes applies a pod's `terminationGracePeriodSeconds` to every kill of the container, a liveness-probe restart included. Celery workers commonly set that period to hours so that a rolling update drains the tasks they are running (the chart's own default is 600 s, production setups go far higher). That turns a liveness restart into an hours-long wait on a worker that stopped answering pings: Celery goes into warm shutdown on the SIGTERM and waits for in-flight tasks, kubelet waits the full grace period before SIGKILL, and nothing consumes the queue in between. We hit this on Airflow 3.3.0 / chart 1.22.0: a Redis failover left three Celery workers connected but not consuming and not answering `celery inspect ping`. The liveness probe fired after 20 minutes, the SIGTERM started a warm shutdown that waited on two hung tasks, and the 3 h pod grace period kept the containers "Running" with no consumer until the pods were deleted by hand. Probes have carried their own `terminationGracePeriodSeconds` since Kubernetes 1.25 (GA), scoped to kills triggered by that probe. This exposes it as `workers.celery.livenessProbe.terminationGracePeriodSeconds`, unset by default, so existing deployments render exactly as before and a deployment that wants a long drain on rollouts can still restart a wedged worker in minutes. Changes: - `templates/workers/worker-deployment.yaml`: render the field under the liveness probe when set. - `values.yaml`: new key, `~` by default, with a comment on when to set it. - `values.schema.json`: new property, integer or null. - `tests/helm_tests/airflow_core/test_worker.py`: unset by default, rendered when set. Newsfragment follows in a second commit once this PR has its number. --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
