anmolxlight commented on code in PR #71939:
URL: https://github.com/apache/airflow/pull/71939#discussion_r4152261518
##########
providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py:
##########
@@ -676,22 +691,21 @@ def _build_track_driver_status_command(self) -> list[str]:
"""
curl_max_wait_time = 30
spark_host = self._connection["master"]
- if spark_host.endswith(":6066"):
- spark_host = spark_host.replace("spark://", "http://";)
- connection_cmd = [
- "/usr/bin/curl",
- "--max-time",
- str(curl_max_wait_time),
- f"{spark_host}/v1/submissions/status/{self._driver_id}",
- ]
- self.log.info(connection_cmd)
-
+ if "spark://" in spark_host:
Review Comment:
Added.
`providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py`
now carries the comment on both branches that switch on it:
```python
spark_host = self._connection["master"]
# spark:// indicates Spark standalone cluster mode
if "spark://" in spark_host:
```
and the same comment on the kill path. Commit c82dd10. Checking the raw
`spark://` prefix instead of the old `:6066` suffix is also what lets a master
URL on the binary RPC port (7077) poll through the REST port.
##########
providers/apache/spark/tests/unit/apache/spark/hooks/test_spark_submit.py:
##########
@@ -164,6 +164,14 @@ def setup_connections(self, create_connection_without_db):
extra='{"deploy-mode": "client"}',
)
)
+ create_connection_without_db(
+ Connection(
+ conn_id="spark_standalone_cluster_rpc_port",
+ conn_type="spark",
+ host="spark://spark-standalone-master:7077",
Review Comment:
Both added.
- `test_build_track_driver_status_command_ha` asserts the emitted command
tries host1 then host2 (`curl ... || curl ...`) for a
`spark://host1:7077,host2:7077` master.
- `test_standalone_curl_command_runs_and_falls_through_to_second_master`
executes the emitted command against a real shell with host1 on a discard port
(curl exits 7) and a local HTTP server on host2, then feeds the output through
`_process_spark_status_log` and asserts `_driver_status == "FINISHED"`. That is
the first-host-fails case end to end, and it also pins that curl's progress
meter cannot splice into the JSON.
- `test_standalone_cluster_ha_kill` covers the same failover for the kill
path.
Commit c82dd10.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
