Re: [PR] fix xcom push not working in `SparkKubernetesOperator` [airflow]
potiuk commented on PR #52051: URL: https://github.com/apache/airflow/pull/52051#issuecomment-4322539265 @kesem0811 This draft PR has had no activity for 3 weeks. Closing to keep the queue clean. You are welcome to reopen and continue when you're ready. If you'd like to pick it back up, please rebase onto the current `main` branch first. --- _Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this [two-stage triage process](https://github.com/apache/airflow/blob/main/contributing-docs/25_maintainer_pr_triage.md#why-the-first-pass-is-automated) so that our maintainers' limited time is spent where it matters most: the conversation with you._ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix xcom push not working in `SparkKubernetesOperator` [airflow]
potiuk closed pull request #52051: fix xcom push not working in `SparkKubernetesOperator` URL: https://github.com/apache/airflow/pull/52051 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix xcom push not working in `SparkKubernetesOperator` [airflow]
potiuk commented on PR #52051: URL: https://github.com/apache/airflow/pull/52051#issuecomment-4177750931 This pull request has had no activity from the author for over 4 weeks. We are converting it to draft to keep the review queue manageable. **@kesem0811**, please mark this PR as ready for review when you are ready to continue working on it. Thank you for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix xcom push not working in `SparkKubernetesOperator` [airflow]
Lee-W commented on PR #52051: URL: https://github.com/apache/airflow/pull/52051#issuecomment-3707596267 will also need your help resolving the conflict. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix xcom push not working in `SparkKubernetesOperator` [airflow]
kunaljubce commented on PR #52051: URL: https://github.com/apache/airflow/pull/52051#issuecomment-3704600481 @kesem0811 Can you also fix the static checks CI error? Should be autofixed by your [prek](https://github.com/apache/airflow/blob/main/contributing-docs/03_contributors_quick_start.rst#configuring-prek) pre-commit hooks, assuming you have them installed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix xcom push not working in `SparkKubernetesOperator` [airflow]
Lee-W commented on code in PR #52051: URL: https://github.com/apache/airflow/pull/52051#discussion_r2414871445 ## providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/utils/xcom_sidecar.py: ## @@ -29,11 +30,13 @@ class PodDefaults: XCOM_MOUNT_PATH = "/airflow/xcom" SIDECAR_CONTAINER_NAME = "airflow-xcom-sidecar" XCOM_CMD = 'trap "exit 0" INT; while true; do sleep 1; done;' -VOLUME_MOUNT = k8s.V1VolumeMount(name="xcom", mount_path=XCOM_MOUNT_PATH) -VOLUME = k8s.V1Volume(name="xcom", empty_dir=k8s.V1EmptyDirVolumeSource()) +VOLUME_MOUNT_NAME = "xcom" +VOLUME_MOUNT = k8s.V1VolumeMount(name=VOLUME_MOUNT_NAME, mount_path=XCOM_MOUNT_PATH) +XCOM_SIDECAR_COMMAND = ["sh", "-c", XCOM_CMD] +VOLUME = k8s.V1Volume(name=VOLUME_MOUNT_NAME, empty_dir=k8s.V1EmptyDirVolumeSource()) Review Comment: ```suggestion XCOM_SIDECAR_COMMAND = ["sh", "-c", XCOM_CMD] VOLUME_MOUNT_NAME = "xcom" VOLUME_MOUNT = k8s.V1VolumeMount(name=VOLUME_MOUNT_NAME, mount_path=XCOM_MOUNT_PATH) VOLUME = k8s.V1Volume(name=VOLUME_MOUNT_NAME, empty_dir=k8s.V1EmptyDirVolumeSource()) ``` let's group similar concepts together ## providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/utils/xcom_sidecar.py: ## @@ -46,14 +49,20 @@ class PodDefaults: def add_xcom_sidecar( -pod: k8s.V1Pod, +pod: k8s.V1Pod | dict, Review Comment: ```suggestion pod: k8s.V1Pod | dict[str, Any], ``` ## providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/utils/xcom_sidecar.py: ## @@ -64,3 +73,27 @@ def add_xcom_sidecar( pod_cp.spec.containers.append(sidecar) return pod_cp + + +def add_sidecar_to_spark_operator_pod_spec( +spec: dict, sidecar_container_image: str | None = None, sidecar_container_resources: dict | None = None +): Review Comment: ```suggestion def add_sidecar_to_spark_operator_pod_spec( spec: dict[str, Any], sidecar_container_image: str | None = None, sidecar_container_resources: dict[str, Any] | None = None ) -> dict[str, Any]: ``` please confirm whether these type are correct. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix xcom push not working in `SparkKubernetesOperator` [airflow]
kesem0811 commented on PR #52051: URL: https://github.com/apache/airflow/pull/52051#issuecomment-737142 the static checks failed because The image /mnt/ci-image-save-v3-linux_amd64-3.10.tar does not exist. I love to make the tests run again? @romsharon98 @eladkal -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix xcom push not working in `SparkKubernetesOperator` [airflow]
eladkal commented on PR #52051: URL: https://github.com/apache/airflow/pull/52051#issuecomment-3161001889 @kesem0811 can you fix the static checks? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
