Himani78116 opened a new pull request, #74501:
URL: https://github.com/apache/airflow/pull/74501
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
## Description
When do_xcom_push=True is used in KubernetesPodOperator, a sidecar container
is added to handle XCom extraction. However, this sidecar often remains in a
Running state even after the base container has completed its work, resulting
in "zombie pods" that linger in the cluster and consume resources.
This PR implements a synchronization mechanism using a shared emptyDir
volume (sentinel pattern).
1. A shared volume named airflow-xcom-sync is added to the Pod specification.
2. This volume is mounted in both the base container and the XCom sidecar
container.
3. The base container signals completion by writing to this shared volume,
allowing the sidecar to detect the exit and terminate voluntarily.
* closes: #58968
## Changes Made:
- Modified KubernetesPodOperator.build_pod_request_obj:
- Added a shared V1Volume named airflow-xcom-sync using emptyDir to the
Pod specification when do_xcom_push is enabled.
- Added a V1VolumeMount for this volume to the base container at
/tmp/airflow-xcom-sync.
- Added a V1VolumeMount for this volume to the XCom sidecar container at
/tmp/airflow-xcom-sync.
- Added test_xcom_sync.py:
- Implemented unit tests to verify that the synchronization volume and its
mounts are correctly added to the Pod spec when do_xcom_push is True and
omitted when False.
- Used mocking for KubernetesHook to ensure tests remain independent of
local cluster configuration.
## Verification Results
- Unit Tests: Added test_xcom_sync_volume_added_when_do_xcom_push_true to
verify the Pod specification correctly includes the synchronization volume and
mounts.
- Manual Verification: Deployed a test Pod in a Kind cluster. Vens from
Running $\rightarrow$ Succeeded once the base containercompletes, confirming
that the sidecar no longer creates zombie pods.
- Static Checks: Passed ruff formatting, mypy type checking, and
## Checklist
- [x] Unit tests added/updated
- [x] Manual verification performed in cluster
- [x] No assert used in production code
- [x] Formatting and linting passed
- [x] Type checking passed
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [x] Yes, Claude Code by Anthropic
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
