GitHub user jzhuge opened a pull request: https://github.com/apache/spark/pull/21709
[SPARK-5152][CORE] Read metrics config file from Hadoop file system ## What changes were proposed in this pull request? Enable metrics config file to be read directly from Hadoop file system. ``` spark-submit --conf spark.metrics.conf=hdfs://<nn>:8020/config/spark-metrics.properties spark-submit --conf spark.metrics.conf=s3a://<bucket>/config/spark-metrics.properties ``` There is no need to distribute the config file if either HDFS or S3A is used. If no URI scheme is specified, assume file://. This ensures backwards compatibility. ## How was this patch tested? Unit tests: - MetricsSystemSuite - MetricsConfigSuite - TaskContextSuite Manual tests: - Local file without URI scheme - Local file with file:// - HDFS file path - S3 file path Author: John Zhuge <jzh...@apache.org> You can merge this pull request into a Git repository by running: $ git pull https://github.com/jzhuge/spark SPARK-5152 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21709.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21709 ---- commit 836ffdec985eac62396005726d5fb49a9e3455a5 Author: John Zhuge <jzhuge@...> Date: 2018-06-12T05:27:56Z [SPARK-5152][CORE] Read metrics config file from Hadoop file system Enable metrics config file to be read directly from Hadoop file system. ``` spark-submit --conf spark.metrics.conf=hdfs://<nn>:8020/config/spark-metrics.properties spark-submit --conf spark.metrics.conf=s3a://<bucket>/config/spark-metrics.properties ``` There is no need to distribute the config file if either HDFS or S3A is used. If no URI scheme is specified, assume file://. This ensures backwards compatibility. Unit tests: - MetricsSystemSuite - MetricsConfigSuite - TaskContextSuite Manual tests: - Local file without URI scheme - Local file with file:// - HDFS file path - S3 file path Author: John Zhuge <jzh...@apache.org> ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org