onlyarnav opened a new pull request, #68543:
URL: https://github.com/apache/airflow/pull/68543
Decouples the three resumable deployment backends (Spark standalone
driver-status tracking, YARN cluster mode, and Kubernetes driver-pod tracking)
in `SparkSubmitOperator`.
### Problem
Previously, each method in the `ResumableJobMixin` implementation of
`SparkSubmitOperator` (`submit_job`, `get_job_status`, `is_job_active`,
`is_job_succeeded`, `poll_until_complete`, `on_kill`) branched inline on the
active deployment backend. This spread backend-specific logic across multiple
methods, making the codebase hard to follow and difficult to extend.
### Solution
1. Introduced a Strategy pattern:
- `SparkSubmitResumableBackend` acts as the abstract base class/interface.
- `YarnSparkSubmitBackend`, `KubernetesSparkSubmitBackend`, and
`StandaloneSparkSubmitBackend` encapsulate backend-specific logic.
2. Added a cached `_resumable_backend` property in `SparkSubmitOperator` to
resolve the backend selection exactly once (lazily on first access).
3. Delegated all the mixin methods in `SparkSubmitOperator` directly to the
active strategy backend, keeping the operator clean.
closes: #68505
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
