[
https://issues.apache.org/jira/browse/SPARK-47232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18099209#comment-18099209
]
Noufal Rijal commented on SPARK-47232:
--------------------------------------
*Reproduced on Spark 4.0.0 — standalone cluster mode on Kubernetes (AKS)*
We hit the same issue running Spark 4.0.0 in standalone cluster mode on AKS
with Istio as the reverse proxy. With {{spark.ui.reverseProxy=true}} and
{{spark.ui.reverseProxyUrl=http://<host>/<prefix>}} set on the master, clicking
tabs (Stages, Storage, Environment) from the driver UI navigates back to the
master UI pages instead of the driver's — the {{/proxy/<app-id>/}} prefix is
missing from the tab links.
*Workaround:* Setting {{spark.ui.proxyBase}} as a SparkConf entry has no effect
on the redirect handler. It must be set as a JVM system property via
{{{}spark.driver.extraJavaOptions{}}}:
{{-Dspark.ui.proxyBase=/proxy/<app-id>}}
This restores correct tab navigation within the driver UI.
Confirmed this still affects Spark 4.0.0 — the issue is not fixed in the latest
major version.
> Spark-UI stages and other tabs not accessible in standalone mode when
> reverse-proxy is enabled
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-47232
> URL: https://issues.apache.org/jira/browse/SPARK-47232
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes, Structured Streaming, Web UI
> Affects Versions: 3.4.2, 3.4.0
> Reporter: Sharad Mishra
> Priority: Major
> Attachments: Screenshot 2024-02-09 at 5.04.49 PM.png, Screenshot
> 2024-02-09 at 5.05.48 PM.png, Screenshot 2024-02-09 at 5.09.28 PM.png,
> Screenshot 2024-02-09 at 5.13.01 PM.png
>
>
> When enabled reverse proxy in master and worker configOptions. We're not able
> to access different tabs available in spark UI e.g.(stages, environment,
> storage etc.)
> We're deploying spark through bitnami helm chart :
> [https://github.com/bitnami/charts/tree/main/bitnami/spark]
> Name and Version
> bitnami/spark - 6.0.0
> What steps will reproduce the bug?
> Kubernetes Version: 1.25
> Spark: 3.4.2
> Helm chart: 6.0.0
> Steps to reproduce:
> After installing the chart Spark Cluster(Master and worker) UI is available
> at:
> !Screenshot 2024-02-09 at 5.04.49 PM.png|width=609,height=235!
> [https://spark.staging.abc.com/]
> We are able to access running application by click on applicationID under
> Running Applications link:
> !Screenshot 2024-02-09 at 5.05.48 PM.png|width=620,height=239!
> We can access spark UI by clicking Application Detail UI:
> We are taken to jobs tab when we click on Application Detail UI
> !Screenshot 2024-02-09 at 5.09.28 PM.png|width=617,height=177!
> URL looks like:
> [https://spark.staging.abc.com/proxy/app-20240208103209-0030/stages/]
> When we click any of the tab from spark UI e.g. stages or environment etc, it
> takes us back to spark cluster UI page
> We noticed that endpoint changes to
> !Screenshot 2024-02-09 at 5.13.01 PM.png|width=617,height=191!
> [https://spark.staging.abc.com/stages/]
> instead of
> [https://spark.staging.abc.com/proxy/app-20240208103209-0030/stages/]
>
> Are you using any custom parameters or values?
> Configurations set in values.yaml
> ```
> master:
> configOptions:
> -Dspark.ui.reverseProxy=true
>
> -Dspark.ui.reverseProxyUrl=[https://spark.staging.abc.com|https://spark.staging.abc.com/]
> worker:
> configOptions:
> -Dspark.ui.reverseProxy=true
>
> -Dspark.ui.reverseProxyUrl=[https://spark.staging.abc.com|https://spark.staging.abc.com/]
> service:
> type: ClusterIP
> ports:
> http: 8080
> https: 443
> cluster: 7077
> ingress:
> enabled: true
> pathType: ImplementationSpecific
> apiVersion: ""
> hostname: spark.staging.abc.com
> ingressClassName: "staging"
> path: /
> ```
>
> What is the expected behavior?
> Expected behaviour is that when I click on stages tab, instead of taking me
> to
> [https://spark.staging.abc.com/stages/]
> it should take me to following URL:
> [https://spark.staging.abc.com/proxy/app-20240208103209-0030/stages/]
> What do you see instead?
> current behaviour is it takes me to URL:
> [https://spark.staging.abc.com/stages/] , which shows spark cluster UI with
> master and worker details
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
