nailo2c commented on code in PR #65991:
URL: https://github.com/apache/airflow/pull/65991#discussion_r3213959603
##########
providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py:
##########
@@ -99,13 +99,26 @@ class SparkSubmitHook(BaseHook, LoggingMixin):
job finishes (on both success and on_kill). Useful for cleaning up
sidecars such
as Istio (e.g. ``["curl -X POST localhost:15020/quitquitquit"]``).
Each command
is executed via the shell; failures produce a warning but do not fail
the task.
+ :param yarn_track_via_application_status: If True (and deploy_mode is YARN
cluster),
+ release the ``spark-submit`` JVM once the application has been
submitted to
+ YARN, then poll ``yarn application -status <appId>`` every
+ ``status_poll_interval`` seconds until the application reaches a final
state.
+ This frees the worker from holding the long-lived submit JVM. Requires
the
+ ``yarn`` CLI on the worker. Cluster-side driver logs should be used
after the
+ switch to polling. Defaults to ``False``.
Review Comment:
nit but acceptable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
