aaron-y-chen commented on code in PR #71949:
URL: https://github.com/apache/airflow/pull/71949#discussion_r3915659822
##########
providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py:
##########
@@ -457,6 +458,16 @@ def _resolve_connection(self) -> dict[str, Any]:
conn_data["keytab"] =
self._create_keytab_path_from_base64_keytab(
base64_keytab, conn_data["principal"]
)
+ # Construct the Standalone Restendpoint
+ if (
+ conn.conn_type == "spark"
+ and conn_data["master"].startswith("spark://")
+ and conn_data["deploy_mode"] == "cluster"
+ ):
+ host = conn_data["master"].replace("spark://", "").strip()
+ conn_data["rest_endpoint"] = (
+
f"{conn_data['rest_scheme']}://{host.split(':')[0]}:{conn_data['rest_port']}"
Review Comment:
I think the HA regression can be demonstrated with a small unit test:
```python
def test_standalone_cluster_kill_preserves_ha_master_list(self,
create_connection_without_db):
create_connection_without_db(
Connection(
conn_id="spark_standalone_cluster_ha",
conn_type="spark",
host="spark://m1:6066,m2:6066",
extra={"deploy-mode": "cluster"},
)
)
hook = SparkSubmitHook(conn_id="spark_standalone_cluster_ha")
hook._driver_id = "driver-123"
assert hook._build_spark_driver_kill_command() == [
"spark-submit",
"--master",
"spark://m1:6066,m2:6066",
"--kill",
"driver-123",
]
```
This passes before this PR because the complete master list is passed to
Spark. On the current head it fails because the generated curl command targets
only m1.
Please preserve that behavior, either by retaining `spark-submit --kill` for
multi-master configurations or by implementing equivalent per-master REST
fallback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
