Re: JMXSink for YARN deployment

Kousuke Saruta Thu, 11 Sep 2014 10:24:50 -0700

Hi, Vladimir

You can see about --files option athttps://spark.apache.org/docs/latest/running-on-yarn.html

If you use --files option like "--files /path-to-metrics.properties",the file is distributed to each executor/driver's working directory, sothey can load the file.


- Kousuke


(2014/09/12 2:16), Vladimir Tretyakov wrote:

Hi, Kousuke,

Can you please explain a bit detailed what do you mean, I am new inSpark, looked athttps://spark.apache.org/docs/latest/submitting-applications.htmlseems there is no '--files' option.

I just have to add '--files /path-to-metrics.properties' ?Undocumented ability?


Thx for answer.

On Thu, Sep 11, 2014 at 5:55 PM, Kousuke Saruta<saru...@oss.nttdata.co.jp <mailto:saru...@oss.nttdata.co.jp>> wrote:


    Hi Vladimir

    How about use --files option with spark-submit?

    - Kousuke


    (2014/09/11 23:43), Vladimir Tretyakov wrote:

    Hi again, yeah , I've tried to use ” spark.metrics.conf” before
    my question in ML, had no  luck:(
    Any other ideas from somebody?
    Seems nobody use metrics in YARN deployment mode.
    How about Mesos? I didn't try but maybe Spark has the same
    difficulties on Mesos?

    PS: Spark is great thing in general, will be nice to see metrics
    in YARN/Mesos mode, not only in Standalone:)


    On Thu, Sep 11, 2014 at 5:25 PM, Shao, Saisai
    <saisai.s...@intel.com <mailto:saisai.s...@intel.com>> wrote:

        I think you can try to use ”spark.metrics.conf” to manually
        specify the path of metrics.properties, but the prerequisite
        is that each container should find this file in their local
        FS because this file is loaded locally.

        Besides I think this might be a kind of workaround, a better
        solution is to fix this by some other solutions.

        Thanks

        Jerry

        *From:*Vladimir Tretyakov
        [mailto:vladimir.tretya...@sematext.com
        <mailto:vladimir.tretya...@sematext.com>]
        *Sent:* Thursday, September 11, 2014 10:08 PM
        *Cc:* user@spark.apache.org <mailto:user@spark.apache.org>
        *Subject:* Re: JMXSink for YARN deployment

        Hi Shao, thx for explanation, any ideas how to fix it? Where
        should I put metrics.properties file?

        On Thu, Sep 11, 2014 at 4:18 PM, Shao, Saisai
        <saisai.s...@intel.com <mailto:saisai.s...@intel.com>> wrote:

        Hi，

        I’m guessing the problem is that driver or executor cannot
        get the metrics.properties configuration file in the yarn
        container, so metrics system cannot load the right sinks.

        Thanks

        Jerry

        *From:*Vladimir Tretyakov
        [mailto:vladimir.tretya...@sematext.com
        <mailto:vladimir.tretya...@sematext.com>]
        *Sent:* Thursday, September 11, 2014 7:30 PM
        *To:* user@spark.apache.org <mailto:user@spark.apache.org>
        *Subject:* JMXSink for YARN deployment

        Hello, we are in Sematext (https://apps.sematext.com/) are
        writing Monitoring tool for Spark and we came across one
        question:

        How to enable JMX metrics for YARN deployment?

        We put "*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink"

        to file $SPARK_HOME/conf/metrics.properties but it doesn't work.

        Everything works in Standalone mode, but not in YARN mode.

        Can somebody help?

        Thx!

        PS: I've found also
        
https://stackoverflow.com/questions/23529404/spark-on-yarn-how-to-send-metrics-to-graphite-sink/25786112
        without answer.

Re: JMXSink for YARN deployment

Reply via email to