nailo2c commented on code in PR #65991:
URL: https://github.com/apache/airflow/pull/65991#discussion_r3315112080
##########
providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py:
##########
@@ -256,6 +308,10 @@ def _resolve_should_track_driver_status(self) -> bool:
"""
return "spark://" in self._connection["master"] and
self._connection["deploy_mode"] == "cluster"
+ def _should_track_yarn_application_via_rm_api(self) -> bool:
+ """Return whether this submit should switch to YARN RM REST API
polling."""
+ return self._yarn_track_via_rm_api and self._is_yarn and
self._connection["deploy_mode"] == "cluster"
Review Comment:
Good point, I'll make this fail fast when `yarn_track_via_rm_api=True` is
used outside YARN cluster mode, so users don't get a silent fallback to the
blocking submit behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
