Hello again, thx for doc, tried different ways with '--files', no luck. Can somebody who already has Spark on YARN try enable JMX metrics sink? Maybe problem in my hands:)
PS, I've also tried to play with 'yarn.nodemanager.local-dirs', no results. On Thu, Sep 11, 2014 at 8:23 PM, Kousuke Saruta <saru...@oss.nttdata.co.jp> wrote: > Hi, Vladimir > > You can see about --files option at > https://spark.apache.org/docs/latest/running-on-yarn.html > > If you use --files option like "--files /path-to-metrics.properties", the > file is distributed to each executor/driver's working directory, so they > can load the file. > > - Kousuke > > > > (2014/09/12 2:16), Vladimir Tretyakov wrote: > > Hi, Kousuke, > > Can you please explain a bit detailed what do you mean, I am new in > Spark, looked at > https://spark.apache.org/docs/latest/submitting-applications.html seems > there is no '--files' option. > > I just have to add '--files /path-to-metrics.properties' ? Undocumented > ability? > > Thx for answer. > > > > On Thu, Sep 11, 2014 at 5:55 PM, Kousuke Saruta <saru...@oss.nttdata.co.jp > > wrote: > >> Hi Vladimir >> >> How about use --files option with spark-submit? >> >> - Kousuke >> >> >> (2014/09/11 23:43), Vladimir Tretyakov wrote: >> >> Hi again, yeah , I've tried to use ” spark.metrics.conf” before my >> question in ML, had no luck:( >> Any other ideas from somebody? >> Seems nobody use metrics in YARN deployment mode. >> How about Mesos? I didn't try but maybe Spark has the same difficulties >> on Mesos? >> >> PS: Spark is great thing in general, will be nice to see metrics in >> YARN/Mesos mode, not only in Standalone:) >> >> >> On Thu, Sep 11, 2014 at 5:25 PM, Shao, Saisai <saisai.s...@intel.com> >> wrote: >> >>> I think you can try to use ” spark.metrics.conf” to manually specify >>> the path of metrics.properties, but the prerequisite is that each container >>> should find this file in their local FS because this file is loaded locally. >>> >>> >>> >>> Besides I think this might be a kind of workaround, a better solution is >>> to fix this by some other solutions. >>> >>> >>> >>> Thanks >>> >>> Jerry >>> >>> >>> >>> *From:* Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com] >>> *Sent:* Thursday, September 11, 2014 10:08 PM >>> *Cc:* user@spark.apache.org >>> *Subject:* Re: JMXSink for YARN deployment >>> >>> >>> >>> Hi Shao, thx for explanation, any ideas how to fix it? Where should I >>> put metrics.properties file? >>> >>> >>> >>> On Thu, Sep 11, 2014 at 4:18 PM, Shao, Saisai <saisai.s...@intel.com> >>> wrote: >>> >>> Hi， >>> >>> >>> >>> I’m guessing the problem is that driver or executor cannot get the >>> metrics.properties configuration file in the yarn container, so metrics >>> system cannot load the right sinks. >>> >>> >>> >>> Thanks >>> >>> Jerry >>> >>> >>> >>> *From:* Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com] >>> *Sent:* Thursday, September 11, 2014 7:30 PM >>> *To:* user@spark.apache.org >>> *Subject:* JMXSink for YARN deployment >>> >>> >>> >>> Hello, we are in Sematext (https://apps.sematext.com/) are writing >>> Monitoring tool for Spark and we came across one question: >>> >>> >>> >>> How to enable JMX metrics for YARN deployment? >>> >>> >>> >>> We put "*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink" >>> >>> to file $SPARK_HOME/conf/metrics.properties but it doesn't work. >>> >>> >>> >>> Everything works in Standalone mode, but not in YARN mode. >>> >>> >>> >>> Can somebody help? >>> >>> >>> >>> Thx! >>> >>> >>> >>> PS: I've found also >>> https://stackoverflow.com/questions/23529404/spark-on-yarn-how-to-send-metrics-to-graphite-sink/25786112 >>> without answer. >>> >>> >>> >> >> >> > >
