anmolxlight opened a new pull request, #71939:
URL: https://github.com/apache/airflow/pull/71939
Fixes #71881
## Problem
In Spark Standalone cluster mode, `SparkSubmitOperator` fails with
`AirflowException: Failed to poll for the driver status 10 times` even though
the Spark job itself completes fine.
A Spark Standalone master exposes two endpoints:
- the **binary RPC port** (default 7077) used for submitting applications
- the **REST API** on `spark.master.rest.port` (default 6066) used for
status/kill requests
The hook only used curl when the configured master URL literally ended in
`:6066`. With the standard setup (master on 7077), it polled via `spark-submit
--status`, whose `RestSubmissionClient` derives its HTTP URL from the master
URL itself — so it tries to make an HTTP request against the binary RPC port
and can never connect. This is also why a retry sometimes "works": if the task
retried after the job already finished, the status request happened to succeed
through a different path.
## Fix
- Add `_get_standalone_rest_base_url()` which derives the REST base URL from
the master host plus the existing `rest-scheme` / `rest-port` connection extras
— the same resolution `_StandaloneSparkSubmitBackend.get_job_status()` (added
in 3.3) already uses.
- `_build_track_driver_status_command()` now polls via curl for any
`spark://` master instead of only `:6066`-suffixed ones.
- Apply the same fix to `_build_spark_driver_kill_command()`: `spark-submit
--kill` goes through the same broken `RestSubmissionClient` path, so standalone
kills now use `DELETE /v1/submissions/kill/{driverId}`.
- YARN/mesos/local masters keep the existing `spark-submit --status/--kill`
behavior.
- Users with non-default REST ports/TLS can set the documented `rest-port` /
`rest-scheme` connection extras.
## Tests
- New regression tests: master on RPC port 7077 must produce curl commands
against the REST port for both status polling and kill.
- Existing `test_build_track_driver_status_command` /
`test_standalone_cluster_process_on_kill` updated for the new kill command
shape.
- Full hook + operator suites pass locally: 205 passed.
- `ruff check` / `ruff format --check` clean; scoped prek run clean except a
pre-existing environment failure (`check-provider-yaml-valid` rejects
AIRFLOW_HOME == checkout dir, fails on clean main too).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
