[jira] [Commented] (SPARK-13258) --conf properties not honored in Mesos cluster mode
[
https://issues.apache.org/jira/browse/SPARK-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15223668#comment-15223668
]
Vidhya Arvind commented on SPARK-13258:
---
hi - wondering if this has been fixed?
> --conf properties not honored in Mesos cluster mode
> ---
>
> Key: SPARK-13258
> URL: https://issues.apache.org/jira/browse/SPARK-13258
> Project: Spark
> Issue Type: Bug
> Components: Mesos
>Affects Versions: 1.6.0
>Reporter: Michael Gummelt
>
> Spark properties set on {{spark-submit}} via the deprecated
> {{SPARK_JAVA_OPTS}} are passed along to the driver, but those set via the
> preferred {{--conf}} are not.
> For example, this results in the URI being fetched in the executor:
> {{SPARK_JAVA_OPTS="-Dspark.mesos.uris=https://raw.githubusercontent.com/mesosphere/spark/master/README.md
> -Dspark.mesos.executor.docker.image=mesosphere/spark:1.6.0"
> ./bin/spark-submit --deploy-mode cluster --master mesos://10.0.78.140:7077
> --class org.apache.spark.examples.SparkPi
> http://downloads.mesosphere.com.s3.amazonaws.com/assets/spark/spark-examples_2.10-1.5.0.jar}}
> This does not:
> {{SPARK_JAVA_OPTS="-Dspark.mesos.executor.docker.image=mesosphere/spark:1.6.0"
> ./bin/spark-submit --deploy-mode cluster --master mesos://10.0.78.140:7077
> --conf
> spark.mesos.uris=https://raw.githubusercontent.com/mesosphere/spark/master/README.md
> --class org.apache.spark.examples.SparkPi
> http://downloads.mesosphere.com.s3.amazonaws.com/assets/spark/spark-examples_2.10-1.5.0.jar}}
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L369
> In the above line of code, you can see that SPARK_JAVA_OPTS is passed along
> to the driver, so those properties take effect.
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L373
> Whereas in this line of code, you see that {{--conf}} variables are set on
> {{SPARK_EXECUTOR_OPTS}}, which AFAICT has absolutely no effect because this
> env var is being set on the driver, not the executor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-5784) Add StatsDSink to MetricsSystem
[
https://issues.apache.org/jira/browse/SPARK-5784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vidhya Arvind updated SPARK-5784:
-
Attachment: statsd.patch
Attaching patch file. Not sure how to get this jira reopened and patch added
Add StatsDSink to MetricsSystem
---
Key: SPARK-5784
URL: https://issues.apache.org/jira/browse/SPARK-5784
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.2.1
Reporter: Ryan Williams
Priority: Minor
Attachments: statsd.patch
[StatsD|https://github.com/etsy/statsd/] is a common wrapper for Graphite; it
would be useful to support sending metrics to StatsD in addition to [the
existing Graphite
support|https://github.com/apache/spark/blob/6a1be026cf37e4c8bf39133dfb4a73f7caedcc26/core/src/main/scala/org/apache/spark/metrics/sink/GraphiteSink.scala].
[readytalk/metrics-statsd|https://github.com/readytalk/metrics-statsd] is a
StatsD adapter for the
[dropwizard/metrics|https://github.com/dropwizard/metrics] library that Spark
uses. The Maven repository at http://dl.bintray.com/readytalk/maven/ serves
{{metrics-statsd}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5784) Add StatsDSink to MetricsSystem
[
https://issues.apache.org/jira/browse/SPARK-5784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572206#comment-14572206
]
Vidhya Arvind commented on SPARK-5784:
--
[~rdub] I tried to add this as package, it's the same problem as Sink and
MetricsSystem are private and are not accessible. I am trying to add this as
part of extras module to do custom assembly. I could build it successfully so
far
Add StatsDSink to MetricsSystem
---
Key: SPARK-5784
URL: https://issues.apache.org/jira/browse/SPARK-5784
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.2.1
Reporter: Ryan Williams
Priority: Minor
[StatsD|https://github.com/etsy/statsd/] is a common wrapper for Graphite; it
would be useful to support sending metrics to StatsD in addition to [the
existing Graphite
support|https://github.com/apache/spark/blob/6a1be026cf37e4c8bf39133dfb4a73f7caedcc26/core/src/main/scala/org/apache/spark/metrics/sink/GraphiteSink.scala].
[readytalk/metrics-statsd|https://github.com/readytalk/metrics-statsd] is a
StatsD adapter for the
[dropwizard/metrics|https://github.com/dropwizard/metrics] library that Spark
uses. The Maven repository at http://dl.bintray.com/readytalk/maven/ serves
{{metrics-statsd}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5784) Add StatsDSink to MetricsSystem
[
https://issues.apache.org/jira/browse/SPARK-5784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568382#comment-14568382
]
Vidhya Arvind commented on SPARK-5784:
--
Hi - checking to see if there are plans to merge this to master?
Add StatsDSink to MetricsSystem
---
Key: SPARK-5784
URL: https://issues.apache.org/jira/browse/SPARK-5784
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.2.1
Reporter: Ryan Williams
Priority: Minor
[StatsD|https://github.com/etsy/statsd/] is a common wrapper for Graphite; it
would be useful to support sending metrics to StatsD in addition to [the
existing Graphite
support|https://github.com/apache/spark/blob/6a1be026cf37e4c8bf39133dfb4a73f7caedcc26/core/src/main/scala/org/apache/spark/metrics/sink/GraphiteSink.scala].
[readytalk/metrics-statsd|https://github.com/readytalk/metrics-statsd] is a
StatsD adapter for the
[dropwizard/metrics|https://github.com/dropwizard/metrics] library that Spark
uses. The Maven repository at http://dl.bintray.com/readytalk/maven/ serves
{{metrics-statsd}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
