Given a session/context, we can get the UI web URL like this: sparkSession.sparkContext.uiWebUrl
This gives me something like http://node-name.cluster-name:4040. If opening this from outside the cluster (ex: my laptop), this redirects via HTTP 302 to something like http://node-name.cluster-name:8088/proxy/redirect/application_1579210019853_0023/. For discussion purposes, call the latter one the "final web URL". Critically, this final URL is active even after the application terminates. The original uiWebUrl (http://node-name.cluster-name:4040) is not available after the application terminates, so one has to have captured the redirect in time, if they want to provide a persistent link to that history server UI entry (ex: for debugging purposes). Is there a way, other than using some HTTP client, to detect what this final URL will be directly from the SparkContext? --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
