hunnguye commented on issue #20949: URL: https://github.com/apache/airflow/issues/20949#issuecomment-5326912630
I would like to revive this issue. We are working with AIrflow 3.3 and a Spark Standalone Cluster deployment. The main issue is, that this kind of deployment has two endpoints. * 7077: the binary endpoint for submitting an application * 6066: the REST endpoint, in this case especially relevant for status polling. https://github.com/apache/airflow/blob/f8b8461e8191f88e72ad8c05b248e0385c21db99/providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py#L671 After a spark submit, the hook tries to poll the status using the method above. However, it conditionally branches depending on whether the port is 6066 or not using either a curl or the spark binaries to poll the job status. If we submit an application using the binary endpoint (7077), it will therefore try to invoke a `spark-submit --master url --status driverid` for polling the status, which will cause the error message as described in the inital issue statement. This is because the spark submit binary internally apperently executes a rest call when requesting the status which requires the endpoint at 6066. Although submitting an application through the REST endpoint to spark should be possible, Airflow only supports the binary approach anyway. Therefore, the defined connection_id has to point to the binary endpoint https://github.com/apache/airflow/blob/f8b8461e8191f88e72ad8c05b248e0385c21db99/providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py#L650 The workaround as other mentioned, is to override the method of the hook to not conditionally branch on port 6066 or use some other logic, to invoke a curl call to the restendpoint (on 6066) even if the master url doesn't mention port 6066. However, I also noticed that with recent spark provider development, this issue has already been partially solved by the spark submit operator. https://github.com/apache/airflow/blob/f8b8461e8191f88e72ad8c05b248e0385c21db99/providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_submit.py#L204 Starting with Airflow 3.3, a task retry would check on job_status first to figure out, whether the previous try actually succeded and also uses a status request (similiar to that of the polling). I believe it contains the logic to create a correct url based on the "master url" even though the master url might point to the binary endpoint. However this logic resides in the Operator. Unfortunately, I'm not familiar with developing and contributing to such a complex project. If maintainers agree with my oservation and see the need to fix this issue (if it even is one, maybe its by design), I'd like to work on a patch with some guidance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
