blcksrx commented on code in PR #61528:
URL: https://github.com/apache/airflow/pull/61528#discussion_r2779635676
##########
providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py:
##########
@@ -265,10 +265,20 @@ def _resolve_connection(self) -> dict[str, Any]:
# Master can be local, yarn, spark://HOST:PORT, mesos://HOST:PORT
and
# k8s://https://<HOST>:<PORT>
conn = self.get_connection(self._conn_id)
- if conn.port:
- conn_data["master"] = f"{conn.host}:{conn.port}"
- else:
+
+ # When connection is created from URI, the scheme (spark://,
k8s://, etc.)
+ # is stored in conn_type, and conn.host contains only the hostname.
+ # When created from UI, conn_type is typically "spark" and
conn.host
+ # may contain the full master URL (e.g., k8s://https://host).
+ if conn.conn_type == "spark" and conn.host and ("://" in conn.host
or not conn.port):
+ # UI-based spark connection where host contains the full
master URL
conn_data["master"] = conn.host
+ else:
+ # Reconstruct URL with conn_type as protocol
+ conn_data["master"] = f"{conn.conn_type}://{conn.host or ''}"
Review Comment:
How? It's vital to set the connection type and connection host based on the
connection provided.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
