[
https://issues.apache.org/jira/browse/SPARK-15487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15600365#comment-15600365
]
Cédric Hernalsteens commented on SPARK-15487:
---------------------------------------------
While it works great to proxy to the slave nodes (running in docker containers
with ports not exposed), there seem to be some inconsistencies in the way the
URI are generated.
I specified
- SPARK_MASTER_OPTS=-Dspark.ui.reverseProxyUrl=http://dev-machine-1/spark/
-Dspark.ui.reverseProxy=true
- SPARK_WORKER_OPTS=-Dspark.ui.reverseProxyUrl=http://dev-machine-1/spark/
-Dspark.ui.reverseProxy=true
Hoping to be able to reverse proxy the whole thing with nginx:
location /spark/ {
proxy_pass http://spark-master:8080/;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
However in the Master webUI, the URI for the slave nodes are
http://dev-machine-1/proxy/... and not http://dev-machine-1/spark/proxy/...
Then I decided to use a nginx directive " location /proxy/ " which is a bit
ugly but then on the Worker webUI, the URI "back to master" is
http://dev-machine-1/spark , apparently taking spark.ui.reverseProxyUrl into
account.
Would someone be able to point to what I'm doing wrong, or confirm this is an
issue?
> Spark Master UI to reverse proxy Application and Workers UI
> -----------------------------------------------------------
>
> Key: SPARK-15487
> URL: https://issues.apache.org/jira/browse/SPARK-15487
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
> Affects Versions: 1.6.0, 1.6.1
> Reporter: Gurvinder
> Assignee: Gurvinder
> Priority: Minor
> Fix For: 2.1.0
>
>
> Currently when running in Standalone mode, Spark UI's link to workers and
> application drivers are pointing to internal/protected network endpoints. So
> to access workers/application UI user's machine has to connect to VPN or need
> to have access to internal network directly.
> Therefore the proposal is to make Spark master UI reverse proxy this
> information back to the user. So only Spark master UI needs to be opened up
> to internet.
> The minimal changes can be done by adding another route e.g.
> http://spark-master.com/target/<endpoint>/ so when request goes to target,
> ProxyServlet kicks in and takes the <endpoint> and forwards the request to it
> and send response back to user.
> More information about discussions for this features can be found on this
> mailing list thread
> http://apache-spark-developers-list.1001551.n3.nabble.com/spark-on-kubernetes-tc17599.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
