amoghrajesh commented on code in PR #68067:
URL: https://github.com/apache/airflow/pull/68067#discussion_r3393352786
##########
providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_submit.py:
##########
@@ -140,6 +145,11 @@ class SparkSubmitOperator(ResumableJobMixin, BaseOperator):
# YARN application ID, K8s driver pod name).
external_id_key = "spark_job_id"
+ # Used only for k8s cluster mode. Caches the pod phase ("Succeeded" /
"Failed") to task_store at the end of
+ # poll_until_complete. On retry, get_job_status reads this before querying
the K8s API
+ # so that a completed job can be identified even after the driver pod is
garbage collected.
+ _K8S_DRIVER_STATUS_KEY = "k8s_driver_status"
Review Comment:
`external_id_key` is a `ResumableJobMixin` interface attribute, it's
lowercase by the mixin contract and matched across all `ResumableJobMixin`
implementations. `_K8S_DRIVER_STATUS_KEY` is a private class constant, where
UPPER_SNAKE_CASE is the Python convention. The different casing is intentional.
[PEP8](https://peps.python.org/pep-0008/#constants) also recommends:
`Constants are usually defined on a module level and written in all capital
letters with underscores separating words.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
