[jira] [Commented] (SPARK-7169) Allow to specify metrics configuration more flexibly
[
https://issues.apache.org/jira/browse/SPARK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744200#comment-14744200
]
Ryan Williams commented on SPARK-7169:
--
[~jlewandowski] I assume [~jerryshao] is referring to the ability to set values
in {{metrics.properties}} that are specific to the driver vs. the executors;
see [these lines in
metrics.properties.template|https://github.com/apache/spark/blob/v1.5.0/conf/metrics.properties.template#L146-L148].
> Allow to specify metrics configuration more flexibly
>
>
> Key: SPARK-7169
> URL: https://issues.apache.org/jira/browse/SPARK-7169
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 1.2.2, 1.3.1
>Reporter: Jacek Lewandowski
>Assignee: Marcelo Vanzin
>Priority: Minor
> Fix For: 1.5.0
>
>
> Metrics are configured in {{metrics.properties}} file. Path to this file is
> specified in {{SparkConf}} at a key {{spark.metrics.conf}}. The property is
> read when {{MetricsSystem}} is created which means, during {{SparkEnv}}
> initialisation.
> h5.Problem
> When the user runs his application he has no way to provide the metrics
> configuration for executors. Although one can specify the path to metrics
> configuration file (1) the path is common for all the nodes and the client
> machine so there is implicit assumption that all the machines has same file
> in the same location, and (2) actually the user needs to copy the file
> manually to the worker nodes because the file is read before the user files
> are populated to the executor local directories. All of this makes it very
> difficult to play with the metrics configuration.
> h5. Proposed solution
> I think that the easiest and the most consistent solution would be to move
> the configuration from a separate file directly to {{SparkConf}}. We may
> prefix all the configuration settings from the metrics configuration by, say
> {{spark.metrics.props}}. For the backward compatibility, these properties
> would be loaded from the specified as it works now. Such a solution doesn't
> change the API so maybe it could be even included in patch release of Spark
> 1.2 and Spark 1.3.
> Appreciate any feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7169) Allow to specify metrics configuration more flexibly
[
https://issues.apache.org/jira/browse/SPARK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593165#comment-14593165
]
Jacek Lewandowski commented on SPARK-7169:
--
[~vanzin] its fix version is set to 1.4.0, however the fix is missing in
branch-1.4, it is present in master.
Allow to specify metrics configuration more flexibly
Key: SPARK-7169
URL: https://issues.apache.org/jira/browse/SPARK-7169
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.2.2, 1.3.1
Reporter: Jacek Lewandowski
Assignee: Marcelo Vanzin
Priority: Minor
Fix For: 1.4.0
Metrics are configured in {{metrics.properties}} file. Path to this file is
specified in {{SparkConf}} at a key {{spark.metrics.conf}}. The property is
read when {{MetricsSystem}} is created which means, during {{SparkEnv}}
initialisation.
h5.Problem
When the user runs his application he has no way to provide the metrics
configuration for executors. Although one can specify the path to metrics
configuration file (1) the path is common for all the nodes and the client
machine so there is implicit assumption that all the machines has same file
in the same location, and (2) actually the user needs to copy the file
manually to the worker nodes because the file is read before the user files
are populated to the executor local directories. All of this makes it very
difficult to play with the metrics configuration.
h5. Proposed solution
I think that the easiest and the most consistent solution would be to move
the configuration from a separate file directly to {{SparkConf}}. We may
prefix all the configuration settings from the metrics configuration by, say
{{spark.metrics.props}}. For the backward compatibility, these properties
would be loaded from the specified as it works now. Such a solution doesn't
change the API so maybe it could be even included in patch release of Spark
1.2 and Spark 1.3.
Appreciate any feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7169) Allow to specify metrics configuration more flexibly
[
https://issues.apache.org/jira/browse/SPARK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568022#comment-14568022
]
Apache Spark commented on SPARK-7169:
-
User 'vanzin' has created a pull request for this issue:
https://github.com/apache/spark/pull/6560
Allow to specify metrics configuration more flexibly
Key: SPARK-7169
URL: https://issues.apache.org/jira/browse/SPARK-7169
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.2.2, 1.3.1
Reporter: Jacek Lewandowski
Priority: Minor
Metrics are configured in {{metrics.properties}} file. Path to this file is
specified in {{SparkConf}} at a key {{spark.metrics.conf}}. The property is
read when {{MetricsSystem}} is created which means, during {{SparkEnv}}
initialisation.
h5.Problem
When the user runs his application he has no way to provide the metrics
configuration for executors. Although one can specify the path to metrics
configuration file (1) the path is common for all the nodes and the client
machine so there is implicit assumption that all the machines has same file
in the same location, and (2) actually the user needs to copy the file
manually to the worker nodes because the file is read before the user files
are populated to the executor local directories. All of this makes it very
difficult to play with the metrics configuration.
h5. Proposed solution
I think that the easiest and the most consistent solution would be to move
the configuration from a separate file directly to {{SparkConf}}. We may
prefix all the configuration settings from the metrics configuration by, say
{{spark.metrics.props}}. For the backward compatibility, these properties
would be loaded from the specified as it works now. Such a solution doesn't
change the API so maybe it could be even included in patch release of Spark
1.2 and Spark 1.3.
Appreciate any feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7169) Allow to specify metrics configuration more flexibly
[
https://issues.apache.org/jira/browse/SPARK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519983#comment-14519983
]
Jacek Lewandowski commented on SPARK-7169:
--
[~jerryshao] unfortunately you are wrong. I would be good if it worked as you
said, but it doesn't.
You cannot use {{--file path-to-metrics-conf}} because, the files provided
this way are copied to the executor after the metrics system is initialised.
Therefore the metrics system cannot use such file. Also, you cannot configure
driver/executor with different parameters. What do you mean by saying that the
metrics system supports such features?
Allow to specify metrics configuration more flexibly
Key: SPARK-7169
URL: https://issues.apache.org/jira/browse/SPARK-7169
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.2.2, 1.3.1
Reporter: Jacek Lewandowski
Priority: Minor
Metrics are configured in {{metrics.properties}} file. Path to this file is
specified in {{SparkConf}} at a key {{spark.metrics.conf}}. The property is
read when {{MetricsSystem}} is created which means, during {{SparkEnv}}
initialisation.
h5.Problem
When the user runs his application he has no way to provide the metrics
configuration for executors. Although one can specify the path to metrics
configuration file (1) the path is common for all the nodes and the client
machine so there is implicit assumption that all the machines has same file
in the same location, and (2) actually the user needs to copy the file
manually to the worker nodes because the file is read before the user files
are populated to the executor local directories. All of this makes it very
difficult to play with the metrics configuration.
h5. Proposed solution
I think that the easiest and the most consistent solution would be to move
the configuration from a separate file directly to {{SparkConf}}. We may
prefix all the configuration settings from the metrics configuration by, say
{{spark.metrics.props}}. For the backward compatibility, these properties
would be loaded from the specified as it works now. Such a solution doesn't
change the API so maybe it could be even included in patch release of Spark
1.2 and Spark 1.3.
Appreciate any feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7169) Allow to specify metrics configuration more flexibly
[
https://issues.apache.org/jira/browse/SPARK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520157#comment-14520157
]
Apache Spark commented on SPARK-7169:
-
User 'jacek-lewandowski' has created a pull request for this issue:
https://github.com/apache/spark/pull/5788
Allow to specify metrics configuration more flexibly
Key: SPARK-7169
URL: https://issues.apache.org/jira/browse/SPARK-7169
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.2.2, 1.3.1
Reporter: Jacek Lewandowski
Priority: Minor
Metrics are configured in {{metrics.properties}} file. Path to this file is
specified in {{SparkConf}} at a key {{spark.metrics.conf}}. The property is
read when {{MetricsSystem}} is created which means, during {{SparkEnv}}
initialisation.
h5.Problem
When the user runs his application he has no way to provide the metrics
configuration for executors. Although one can specify the path to metrics
configuration file (1) the path is common for all the nodes and the client
machine so there is implicit assumption that all the machines has same file
in the same location, and (2) actually the user needs to copy the file
manually to the worker nodes because the file is read before the user files
are populated to the executor local directories. All of this makes it very
difficult to play with the metrics configuration.
h5. Proposed solution
I think that the easiest and the most consistent solution would be to move
the configuration from a separate file directly to {{SparkConf}}. We may
prefix all the configuration settings from the metrics configuration by, say
{{spark.metrics.props}}. For the backward compatibility, these properties
would be loaded from the specified as it works now. Such a solution doesn't
change the API so maybe it could be even included in patch release of Spark
1.2 and Spark 1.3.
Appreciate any feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7169) Allow to specify metrics configuration more flexibly
[
https://issues.apache.org/jira/browse/SPARK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520170#comment-14520170
]
Jacek Lewandowski commented on SPARK-7169:
--
https://github.com/apache/spark/pull/5788
Hey [~joshrosen], could you take a look at my PR? It is very straightforward.
Allow to specify metrics configuration more flexibly
Key: SPARK-7169
URL: https://issues.apache.org/jira/browse/SPARK-7169
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.2.2, 1.3.1
Reporter: Jacek Lewandowski
Priority: Minor
Metrics are configured in {{metrics.properties}} file. Path to this file is
specified in {{SparkConf}} at a key {{spark.metrics.conf}}. The property is
read when {{MetricsSystem}} is created which means, during {{SparkEnv}}
initialisation.
h5.Problem
When the user runs his application he has no way to provide the metrics
configuration for executors. Although one can specify the path to metrics
configuration file (1) the path is common for all the nodes and the client
machine so there is implicit assumption that all the machines has same file
in the same location, and (2) actually the user needs to copy the file
manually to the worker nodes because the file is read before the user files
are populated to the executor local directories. All of this makes it very
difficult to play with the metrics configuration.
h5. Proposed solution
I think that the easiest and the most consistent solution would be to move
the configuration from a separate file directly to {{SparkConf}}. We may
prefix all the configuration settings from the metrics configuration by, say
{{spark.metrics.props}}. For the backward compatibility, these properties
would be loaded from the specified as it works now. Such a solution doesn't
change the API so maybe it could be even included in patch release of Spark
1.2 and Spark 1.3.
Appreciate any feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7169) Allow to specify metrics configuration more flexibly
[
https://issues.apache.org/jira/browse/SPARK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518508#comment-14518508
]
Saisai Shao commented on SPARK-7169:
Hi [~jlewandowski], regard to your second problem, I think you don't have to
copy the metrics configuration file manually to every machine one by one, you
could use spark-submit --file path/to/your/metrics_properties to transfer your
configuration to each executor/container.
And for the first problem, is it a big problem that all the configuration files
need to be in the same directory? I think lot's of Spark as well as Hadoop conf
file has such requirement. But you could configure driver/executor with
different parameters in conf file, since MetricsSystem supports such features.
Yes I think current metrics configuration may not so easy to use, any
improvement is greatly appreciated :).
Allow to specify metrics configuration more flexibly
Key: SPARK-7169
URL: https://issues.apache.org/jira/browse/SPARK-7169
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.2.2, 1.3.1
Reporter: Jacek Lewandowski
Priority: Minor
Metrics are configured in {{metrics.properties}} file. Path to this file is
specified in {{SparkConf}} at a key {{spark.metrics.conf}}. The property is
read when {{MetricsSystem}} is created which means, during {{SparkEnv}}
initialisation.
h5.Problem
When the user runs his application he has no way to provide the metrics
configuration for executors. Although one can specify the path to metrics
configuration file (1) the path is common for all the nodes and the client
machine so there is implicit assumption that all the machines has same file
in the same location, and (2) actually the user needs to copy the file
manually to the worker nodes because the file is read before the user files
are populated to the executor local directories. All of this makes it very
difficult to play with the metrics configuration.
h5. Proposed solution
I think that the easiest and the most consistent solution would be to move
the configuration from a separate file directly to {{SparkConf}}. We may
prefix all the configuration settings from the metrics configuration by, say
{{spark.metrics.props}}. For the backward compatibility, these properties
would be loaded from the specified as it works now. Such a solution doesn't
change the API so maybe it could be even included in patch release of Spark
1.2 and Spark 1.3.
Appreciate any feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7169) Allow to specify metrics configuration more flexibly
[
https://issues.apache.org/jira/browse/SPARK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14514289#comment-14514289
]
Sean Owen commented on SPARK-7169:
--
Just go ahead with a PR, or further comments or design here. I think you want
to get buy in first.
Allow to specify metrics configuration more flexibly
Key: SPARK-7169
URL: https://issues.apache.org/jira/browse/SPARK-7169
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.2.2, 1.3.1, 1.4.0
Reporter: Jacek Lewandowski
Metrics are configured in {{metrics.properties}} file. Path to this file is
specified in {{SparkConf}} at a key {{spark.metrics.conf}}. The property is
read when {{MetricsSystem}} is created which means, during {{SparkEnv}}
initialisation.
h5.Problem
When the user runs his application he has no way to provide the metrics
configuration for executors. Although one can specify the path to metrics
configuration file (1) the path is common for all the nodes and the client
machine so there is implicit assumption that all the machines has same file
in the same location, and (2) actually the user needs to copy the file
manually to the worker nodes because the file is read before the user files
are populated to the executor local directories. All of this makes it very
difficult to play with the metrics configuration.
h5. Proposed solution
I think that the easiest and the most consistent solution would be to move
the configuration from a separate file directly to {{SparkConf}}. We may
prefix all the configuration settings from the metrics configuration by, say
{{spark.metrics.props}}. For the backward compatibility, these properties
would be loaded from the specified as it works now. Such a solution doesn't
change the API so maybe it could be even included in patch release of Spark
1.2 and Spark 1.3.
Appreciate any feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7169) Allow to specify metrics configuration more flexibly
[
https://issues.apache.org/jira/browse/SPARK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14514334#comment-14514334
]
Jacek Lewandowski commented on SPARK-7169:
--
Ok, thanks [~srowen]
Allow to specify metrics configuration more flexibly
Key: SPARK-7169
URL: https://issues.apache.org/jira/browse/SPARK-7169
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.2.2, 1.3.1
Reporter: Jacek Lewandowski
Priority: Minor
Metrics are configured in {{metrics.properties}} file. Path to this file is
specified in {{SparkConf}} at a key {{spark.metrics.conf}}. The property is
read when {{MetricsSystem}} is created which means, during {{SparkEnv}}
initialisation.
h5.Problem
When the user runs his application he has no way to provide the metrics
configuration for executors. Although one can specify the path to metrics
configuration file (1) the path is common for all the nodes and the client
machine so there is implicit assumption that all the machines has same file
in the same location, and (2) actually the user needs to copy the file
manually to the worker nodes because the file is read before the user files
are populated to the executor local directories. All of this makes it very
difficult to play with the metrics configuration.
h5. Proposed solution
I think that the easiest and the most consistent solution would be to move
the configuration from a separate file directly to {{SparkConf}}. We may
prefix all the configuration settings from the metrics configuration by, say
{{spark.metrics.props}}. For the backward compatibility, these properties
would be loaded from the specified as it works now. Such a solution doesn't
change the API so maybe it could be even included in patch release of Spark
1.2 and Spark 1.3.
Appreciate any feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7169) Allow to specify metrics configuration more flexibly
[
https://issues.apache.org/jira/browse/SPARK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14514282#comment-14514282
]
Jacek Lewandowski commented on SPARK-7169:
--
I'd like to start working on this, could someone assign me to this ticket?
Allow to specify metrics configuration more flexibly
Key: SPARK-7169
URL: https://issues.apache.org/jira/browse/SPARK-7169
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.2.2, 1.3.1, 1.4.0
Reporter: Jacek Lewandowski
Metrics are configured in {{metrics.properties}} file. Path to this file is
specified in {{SparkConf}} at a key {{spark.metrics.conf}}. The property is
read when {{MetricsSystem}} is created which means, during {{SparkEnv}}
initialisation.
h5.Problem
When the user runs his application he has no way to provide the metrics
configuration for executors. Although one can specify the path to metrics
configuration file (1) the path is common for all the nodes and the client
machine so there is implicit assumption that all the machines has same file
in the same location, and (2) actually the user needs to copy the file
manually to the worker nodes because the file is read before the user files
are populated to the executor local directories. All of this makes it very
difficult to play with the metrics configuration.
h5. Proposed solution
I think that the easiest and the most consistent solution would be to move
the configuration from a separate file directly to {{SparkConf}}. We may
prefix all the configuration settings from the metrics configuration by, say
{{spark.metrics.props}}. For the backward compatibility, these properties
would be loaded from the specified as it works now. Such a solution doesn't
change the API so maybe it could be even included in patch release of Spark
1.2 and Spark 1.3.
Appreciate any feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
