[GitHub] spark issue #21709: [SPARK-5152][CORE] Read metrics config file from Hadoop ...
Github user jzhuge commented on the issue: https://github.com/apache/spark/pull/21709 Not quite there with SparkConf. We already uses option `--properties-file` for a different purpose and spark-submit does not allow multiple occurrences of `--properties-file`. How about the enhancement to allow multiple occurrences of `--properties-file`? That will solve this issue and serve other use cases as well. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21709: [SPARK-5152][CORE] Read metrics config file from Hadoop ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/21709 From my point, I don't see a special reason that it cannot be achieved by SparkConf. Also you can specify different local metrics conf file for different users/purpose, I don't think it is different from putting it on HDFS. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21709: [SPARK-5152][CORE] Read metrics config file from Hadoop ...
Github user jzhuge commented on the issue: https://github.com/apache/spark/pull/21709 Thx @jerryshao for looking at the PR. Compared to spark.metrics.conf.* in SparkConf, a metrics config file can group the properties together, separate from the rest of the properties. In my case, there are 10 properties to config. It is also easier to swap out the config file for different users or different purposes, especially in self-serving environments. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21709: [SPARK-5152][CORE] Read metrics config file from Hadoop ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/21709 If you want metrics conf to be centralized without needing to put it on to different nodes, you can set it through `SparkConf` with prefix "spark.metrics.conf.", MetricsSystem also supports getting configurations from `SparkConf`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21709: [SPARK-5152][CORE] Read metrics config file from Hadoop ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/21709 Hi @jzhuge what is the purpose of supporting reading metrics conf from HDFS/S3? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21709: [SPARK-5152][CORE] Read metrics config file from Hadoop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21709 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21709: [SPARK-5152][CORE] Read metrics config file from Hadoop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21709 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21709: [SPARK-5152][CORE] Read metrics config file from Hadoop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21709 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
