Re: JMXSink for YARN deployment

[Kousuke Saruta]

Hi Vladimir

How about use --files option with spark-submit?

- Kousuke

(2014/09/11 23:43), Vladimir Tretyakov wrote:

Hi again, yeah , I've tried to use ” spark.metrics.conf” before my question in ML, had no luck:(

Any other ideas from somebody?
Seems nobody use metrics in YARN deployment mode.

How about Mesos? I didn't try but maybe Spark has the same difficulties on Mesos?

PS: Spark is great thing in general, will be nice to see metrics in YARN/Mesos mode, not only in Standalone:)

On Thu, Sep 11, 2014 at 5:25 PM, Shao, Saisai <saisai.s...@intel.com <mailto:saisai.s...@intel.com>> wrote:

    I think you can try to use ”spark.metrics.conf” to manually
    specify the path of metrics.properties, but the prerequisite is
    that each container should find this file in their local FS
    because this file is loaded locally.

    Besides I think this might be a kind of workaround, a better
    solution is to fix this by some other solutions.

    Thanks

    Jerry

    *From:*Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com
    <mailto:vladimir.tretya...@sematext.com>]
    *Sent:* Thursday, September 11, 2014 10:08 PM
    *Cc:* user@spark.apache.org <mailto:user@spark.apache.org>
    *Subject:* Re: JMXSink for YARN deployment

    Hi Shao, thx for explanation, any ideas how to fix it? Where
    should I put metrics.properties file?

    On Thu, Sep 11, 2014 at 4:18 PM, Shao, Saisai
    <saisai.s...@intel.com <mailto:saisai.s...@intel.com>> wrote:

    Hi，

    I’m guessing the problem is that driver or executor cannot get the
    metrics.properties configuration file in the yarn container, so
    metrics system cannot load the right sinks.

    Thanks

    Jerry

    *From:*Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com
    <mailto:vladimir.tretya...@sematext.com>]
    *Sent:* Thursday, September 11, 2014 7:30 PM
    *To:* user@spark.apache.org <mailto:user@spark.apache.org>
    *Subject:* JMXSink for YARN deployment

    Hello, we are in Sematext (https://apps.sematext.com/) are writing
    Monitoring tool for Spark and we came across one question:

    How to enable JMX metrics for YARN deployment?

    We put "*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink"

    to file $SPARK_HOME/conf/metrics.properties but it doesn't work.

    Everything works in Standalone mode, but not in YARN mode.

    Can somebody help?

    Thx!

    PS: I've found also
    
https://stackoverflow.com/questions/23529404/spark-on-yarn-how-to-send-metrics-to-graphite-sink/25786112
    without answer.