hunnguye commented on code in PR #71949:
URL: https://github.com/apache/airflow/pull/71949#discussion_r3846871037
##########
providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py:
##########
@@ -457,6 +458,19 @@ def _resolve_connection(self) -> dict[str, Any]:
conn_data["keytab"] =
self._create_keytab_path_from_base64_keytab(
base64_keytab, conn_data["principal"]
)
+ # Construct the Standalone Restendpoint
+ if (
+ conn.conn_type == "spark"
+ and conn_data["master"].startswith("spark://")
+ and conn_data["deploy_mode"] == "cluster"
+ ):
+ if conn_data["master"].endswith("7077"):
Review Comment:
I wasn’t sure how much I could /should change, so I opted for a
minimal-change PR where the changes would only apply to this specific case when
the host is targeting the RPC endpoint 7077.
Especially the logic for the 6066 branch I took from the original code
snippet found in the _build_track_driver_status_command
(https://github.com/apache/airflow/pull/71949/changes/BASE..7efaf2dca9a88da31e1758622e3a6be83b70b90f#diff-0babb37b3dc9c7b8e967f3c17d012538f06e28f2f8f4cc66d2a8d9a69bfed115L680)
So, should I build the rest_endpoint using rest_scheme and rest_port, and
use that endpoint for both status polling and driver kills, effectively
replacing the current spark-submit --status / --kill commands?
Since this issue has been open for quite a long time
(https://github.com/apache/airflow/issues/20949), I was concerned that
introducing broader or potentially breaking changes might affect users who have
already found workarounds for the current behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
