GitHub user HeartSaVioR opened a pull request: https://github.com/apache/spark/pull/23260
[SPARK-26311][YARN] New feature: custom log URL for stdout/stderr ## What changes were proposed in this pull request? This patch proposes adding a new configuration on YARN mode: custom log URL. This will enable end users to point application logs to external log service which enables to serve logs when NodeManager becomes unavailable. Some pre-defined patterns are available for custom log URL to specify them like path variables. ## How was this patch tested? Manual test. Below run changes executor log URLs in UI pages. ``` ./bin/spark-submit --conf spark.yarn.custom.log.url="{{HttpScheme}}{{NodeHttpAddress}}/test/cluster/{{ClusterId}}/container/{{ContainerId}}/user/{{User}}/filename/{{FileName}}" --class org.apache.spark.examples.SparkPi examples/jars/spark-examples_2.11-<spark_version>.jar ``` Example of stdout log url is below: `http://node-address:node-port`/test/cluster/`workload1`/container/`container_e08_1542798098040_0012_01_000002`/user/`spark`/filename/`stdout` You can merge this pull request into a Git repository by running: $ git pull https://github.com/HeartSaVioR/spark SPARK-26311 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23260.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23260 ---- commit 65cc6a32729cccba340f66c766c7255be4d7f356 Author: Jungtaek Lim (HeartSaVioR) <kabhwan@...> Date: 2018-12-08T06:32:46Z [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org