[ https://issues.apache.org/jira/browse/SPARK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated SPARK-7169: ----------------------------- Priority: Minor (was: Major) Affects Version/s: (was: 1.4.0) > Allow to specify metrics configuration more flexibly > ---------------------------------------------------- > > Key: SPARK-7169 > URL: https://issues.apache.org/jira/browse/SPARK-7169 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 1.2.2, 1.3.1 > Reporter: Jacek Lewandowski > Priority: Minor > > Metrics are configured in {{metrics.properties}} file. Path to this file is > specified in {{SparkConf}} at a key {{spark.metrics.conf}}. The property is > read when {{MetricsSystem}} is created which means, during {{SparkEnv}} > initialisation. > h5.Problem > When the user runs his application he has no way to provide the metrics > configuration for executors. Although one can specify the path to metrics > configuration file (1) the path is common for all the nodes and the client > machine so there is implicit assumption that all the machines has same file > in the same location, and (2) actually the user needs to copy the file > manually to the worker nodes because the file is read before the user files > are populated to the executor local directories. All of this makes it very > difficult to play with the metrics configuration. > h5. Proposed solution > I think that the easiest and the most consistent solution would be to move > the configuration from a separate file directly to {{SparkConf}}. We may > prefix all the configuration settings from the metrics configuration by, say > {{spark.metrics.props}}. For the backward compatibility, these properties > would be loaded from the specified as it works now. Such a solution doesn't > change the API so maybe it could be even included in patch release of Spark > 1.2 and Spark 1.3. > Appreciate any feedback. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org