Vamsi-klu opened a new pull request, #71661: URL: https://github.com/apache/airflow/pull/71661
## Problem #68679 moved K8s/YARN submit tracking into strategy backends. Those backends have to read and mutate `SparkSubmitHook` state that is still private: `_conf`, `_kubernetes_driver_pod`, `_yarn_application_id`, `_poll_k8s_driver_via_api`, `_kill_yarn_application`. Closed draft #71059 over-promoted about 10 members. This is the narrower follow-up. ## What I did I promoted only the members the backends actually use: - `conf` — live dict, not a copy, so in-place key writes still work - `kubernetes_driver_pod` — get/set - `yarn_application_id` — get only - `poll_k8s_driver_via_api` - `kill_yarn_application` (Yarn `on_kill` has the same leak) Private names stay as the backing store, so existing hook tests that set `_yarn_application_id` / `_kubernetes_driver_pod` keep working. I switched the K8s and YARN operator backends to the public names. Out of scope, on purpose: `_connection`, `_start_yarn_application_status_tracking`, `_run_post_submit_commands`, `_driver_id` / `_driver_status`. ## How I did it Properties and a rename on `SparkSubmitHook`, then the same names in `_KubernetesSparkSubmitBackend` and `_YarnSparkSubmitBackend`. Operator MagicMock fixtures now use the public names so `.conf` / `.yarn_application_id` are not accidental new MagicMocks. ## Impact Submit backends no longer reach through private attributes. External subclasses that need the YARN kill or K8s poll can call the public methods. Hook internals and tracking behavior are unchanged. ## Testing Hook tests: - `test_conf_mutation_writes_through` - `test_kubernetes_driver_pod_writes_through` - `test_yarn_application_id_is_get_only` (assignment raises `AttributeError`) - existing K8s poll / YARN kill tests retargeted to the public names Operator tests updated so MagicMock fixtures hit the public surface, including `on_kill` → `kill_yarn_application`. ``` pytest providers/apache/spark/tests/unit/apache/spark/hooks/test_spark_submit.py providers/apache/spark/tests/unit/apache/spark/operators/test_spark_submit.py ``` closes: #71058 --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes (please specify the tool below) Generated-by: Cursor following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) Made with [Cursor](https://cursor.com) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
