Re: JMXSink for YARN deployment

[Kyle Lin]

Hello there


I run both driver and master on the same node, so I got "Port already in
use" exception.

Is there any solution to set different port for each component?

Kyle


2015-12-05 5:54 GMT+08:00 spearson23 :

> Run "spark-submit --help" to see all available options.
>
> To get JMX to work you need to:
>
> spark-submit --driver-java-options "-Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.port=JMX_PORT" --conf
> spark.metrics.conf=metrics.properties --class 'CLASS_NAME' --master
> yarn-cluster --files /PATH/TO/metrics.properties /PATH/TO/JAR.FILE
>
>
> This will run JMX on the driver node on or "JMX_PORT".  Note that the
> driver
> node and the YARN master node are not the same, you'll have to look where
> spark put the driver node and then connect there.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/JMXSink-for-YARN-deployment-tp13958p25572.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>

Re: JMXSink for YARN deployment

[Kyle Lin]

Hello guys

I got a solution.

I set -Dcom.sun.management.jmxremote.port=0 to let system assign a unused
port.

Kyle

2016-01-12 16:54 GMT+08:00 Kyle Lin :

> Hello there
>
>
> I run both driver and master on the same node, so I got "Port already in
> use" exception.
>
> Is there any solution to set different port for each component?
>
> Kyle
>
>
> 2015-12-05 5:54 GMT+08:00 spearson23 :
>
>> Run "spark-submit --help" to see all available options.
>>
>> To get JMX to work you need to:
>>
>> spark-submit --driver-java-options "-Dcom.sun.management.jmxremote
>> -Dcom.sun.management.jmxremote.authenticate=false
>> -Dcom.sun.management.jmxremote.ssl=false
>> -Dcom.sun.management.jmxremote.port=JMX_PORT" --conf
>> spark.metrics.conf=metrics.properties --class 'CLASS_NAME' --master
>> yarn-cluster --files /PATH/TO/metrics.properties /PATH/TO/JAR.FILE
>>
>>
>> This will run JMX on the driver node on or "JMX_PORT".  Note that the
>> driver
>> node and the YARN master node are not the same, you'll have to look where
>> spark put the driver node and then connect there.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/JMXSink-for-YARN-deployment-tp13958p25572.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>>
>

Re: JMXSink for YARN deployment

[spearson23]

We use a metrics.property file on YARN by submitting applications like this:

spark-submit --conf spark.metrics.conf=metrics.properties --class CLASS_NAME
--master yarn-cluster --files /PATH/TO/metrics.properties /PATH/TO/CODE.JAR
/PATH/TO/CONFIG.FILE APP_NAME




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/JMXSink-for-YARN-deployment-tp13958p25570.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: JMXSink for YARN deployment

[spearson23]

Run "spark-submit --help" to see all available options.

To get JMX to work you need to:

spark-submit --driver-java-options "-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=JMX_PORT" --conf
spark.metrics.conf=metrics.properties --class 'CLASS_NAME' --master
yarn-cluster --files /PATH/TO/metrics.properties /PATH/TO/JAR.FILE


This will run JMX on the driver node on or "JMX_PORT".  Note that the driver
node and the YARN master node are not the same, you'll have to look where
spark put the driver node and then connect there.




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/JMXSink-for-YARN-deployment-tp13958p25572.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: JMXSink for YARN deployment

[Otis Gospodnetic]

Hi,

Jerry said I'm guessing, so maybe the thing to try is to check if his
guess is correct.

What about running sudo lsof | grep metrics.properties ?  I imagine you
should be able to see it if the file was found and read.  If Jerry is
right, then I think you will NOT see it.

Next, how about trying some bogus value in metrics.properties, like *.sink.
jmx.class=org.apache.spark.metrics.sink.*BUGUSSink*?  If the file is being
read then specifying such bogus value should make something log an error or
throw exception at start, I assume.  If you don't see this then maybe this
file is not being read at all.

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr  Elasticsearch Support * http://sematext.com/



On Thu, Sep 11, 2014 at 9:18 AM, Shao, Saisai saisai.s...@intel.com wrote:

  Hi，



 I’m guessing the problem is that driver or executor cannot get the
 metrics.properties configuration file in the yarn container, so metrics
 system cannot load the right sinks.



 Thanks

 Jerry



 *From:* Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com]
 *Sent:* Thursday, September 11, 2014 7:30 PM
 *To:* user@spark.apache.org
 *Subject:* JMXSink for YARN deployment



 Hello, we are in Sematext (https://apps.sematext.com/) are writing
 Monitoring tool for Spark and we came across one question:



 How to enable JMX metrics for YARN deployment?



 We put *.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink

 to file $SPARK_HOME/conf/metrics.properties but it doesn't work.



 Everything works in Standalone mode, but not in YARN mode.



 Can somebody help?



 Thx!



 PS: I've found also
 https://stackoverflow.com/questions/23529404/spark-on-yarn-how-to-send-metrics-to-graphite-sink/25786112
 without answer.

RE: JMXSink for YARN deployment

[Shao, Saisai]

Hi，

I’m guessing the problem is that driver or executor cannot get the 
metrics.properties configuration file in the yarn container, so metrics system 
cannot load the right sinks.

Thanks
Jerry

From: Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com]
Sent: Thursday, September 11, 2014 7:30 PM
To: user@spark.apache.org
Subject: JMXSink for YARN deployment

Hello, we are in Sematext (https://apps.sematext.com/) are writing Monitoring 
tool for Spark and we came across one question:

How to enable JMX metrics for YARN deployment?

We put *.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink
to file $SPARK_HOME/conf/metrics.properties but it doesn't work.

Everything works in Standalone mode, but not in YARN mode.

Can somebody help?

Thx!

PS: I've found also 
https://stackoverflow.com/questions/23529404/spark-on-yarn-how-to-send-metrics-to-graphite-sink/25786112
 without answer.

Re: JMXSink for YARN deployment

[Vladimir Tretyakov]

Hi Shao, thx for explanation, any ideas how to fix it? Where should I put
metrics.properties file?

On Thu, Sep 11, 2014 at 4:18 PM, Shao, Saisai saisai.s...@intel.com wrote:

  Hi，



 I’m guessing the problem is that driver or executor cannot get the
 metrics.properties configuration file in the yarn container, so metrics
 system cannot load the right sinks.



 Thanks

 Jerry



 *From:* Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com]
 *Sent:* Thursday, September 11, 2014 7:30 PM
 *To:* user@spark.apache.org
 *Subject:* JMXSink for YARN deployment



 Hello, we are in Sematext (https://apps.sematext.com/) are writing
 Monitoring tool for Spark and we came across one question:



 How to enable JMX metrics for YARN deployment?



 We put *.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink

 to file $SPARK_HOME/conf/metrics.properties but it doesn't work.



 Everything works in Standalone mode, but not in YARN mode.



 Can somebody help?



 Thx!



 PS: I've found also
 https://stackoverflow.com/questions/23529404/spark-on-yarn-how-to-send-metrics-to-graphite-sink/25786112
 without answer.

RE: JMXSink for YARN deployment

[Shao, Saisai]

I think you can try to use ” spark.metrics.conf” to manually specify the path 
of metrics.properties, but the prerequisite is that each container should find 
this file in their local FS because this file is loaded locally.

Besides I think this might be a kind of workaround, a better solution is to fix 
this by some other solutions.

Thanks
Jerry

From: Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com]
Sent: Thursday, September 11, 2014 10:08 PM
Cc: user@spark.apache.org
Subject: Re: JMXSink for YARN deployment

Hi Shao, thx for explanation, any ideas how to fix it? Where should I put 
metrics.properties file?

On Thu, Sep 11, 2014 at 4:18 PM, Shao, Saisai 
saisai.s...@intel.commailto:saisai.s...@intel.com wrote:
Hi，

I’m guessing the problem is that driver or executor cannot get the 
metrics.properties configuration file in the yarn container, so metrics system 
cannot load the right sinks.

Thanks
Jerry

From: Vladimir Tretyakov 
[mailto:vladimir.tretya...@sematext.commailto:vladimir.tretya...@sematext.com]
Sent: Thursday, September 11, 2014 7:30 PM
To: user@spark.apache.orgmailto:user@spark.apache.org
Subject: JMXSink for YARN deployment

Hello, we are in Sematext (https://apps.sematext.com/) are writing Monitoring 
tool for Spark and we came across one question:

How to enable JMX metrics for YARN deployment?

We put *.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink
to file $SPARK_HOME/conf/metrics.properties but it doesn't work.

Everything works in Standalone mode, but not in YARN mode.

Can somebody help?

Thx!

PS: I've found also 
https://stackoverflow.com/questions/23529404/spark-on-yarn-how-to-send-metrics-to-graphite-sink/25786112
 without answer.

Re: JMXSink for YARN deployment

[Vladimir Tretyakov]

Hi again, yeah , I've tried to use ” spark.metrics.conf” before my question
in ML, had no  luck:(
Any other ideas from somebody?
Seems nobody use metrics in YARN deployment mode.
How about Mesos? I didn't try but maybe Spark has the same difficulties on
Mesos?

PS: Spark is great thing in general, will be nice to see metrics in
YARN/Mesos mode, not only in Standalone:)


On Thu, Sep 11, 2014 at 5:25 PM, Shao, Saisai saisai.s...@intel.com wrote:

  I think you can try to use ” spark.metrics.conf” to manually specify the
 path of metrics.properties, but the prerequisite is that each container
 should find this file in their local FS because this file is loaded locally.



 Besides I think this might be a kind of workaround, a better solution is
 to fix this by some other solutions.



 Thanks

 Jerry



 *From:* Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com]
 *Sent:* Thursday, September 11, 2014 10:08 PM
 *Cc:* user@spark.apache.org
 *Subject:* Re: JMXSink for YARN deployment



 Hi Shao, thx for explanation, any ideas how to fix it? Where should I put
 metrics.properties file?



 On Thu, Sep 11, 2014 at 4:18 PM, Shao, Saisai saisai.s...@intel.com
 wrote:

 Hi，



 I’m guessing the problem is that driver or executor cannot get the
 metrics.properties configuration file in the yarn container, so metrics
 system cannot load the right sinks.



 Thanks

 Jerry



 *From:* Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com]
 *Sent:* Thursday, September 11, 2014 7:30 PM
 *To:* user@spark.apache.org
 *Subject:* JMXSink for YARN deployment



 Hello, we are in Sematext (https://apps.sematext.com/) are writing
 Monitoring tool for Spark and we came across one question:



 How to enable JMX metrics for YARN deployment?



 We put *.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink

 to file $SPARK_HOME/conf/metrics.properties but it doesn't work.



 Everything works in Standalone mode, but not in YARN mode.



 Can somebody help?



 Thx!



 PS: I've found also
 https://stackoverflow.com/questions/23529404/spark-on-yarn-how-to-send-metrics-to-graphite-sink/25786112
 without answer.

Re: JMXSink for YARN deployment

[Kousuke Saruta]

Hi Vladimir

How about use --files option with spark-submit?

- Kousuke

(2014/09/11 23:43), Vladimir Tretyakov wrote:
Hi again, yeah , I've tried to use ” spark.metrics.conf” before my 
question in ML, had no  luck:(

Any other ideas from somebody?
Seems nobody use metrics in YARN deployment mode.
How about Mesos? I didn't try but maybe Spark has the same 
difficulties on Mesos?


PS: Spark is great thing in general, will be nice to see metrics in 
YARN/Mesos mode, not only in Standalone:)



On Thu, Sep 11, 2014 at 5:25 PM, Shao, Saisai saisai.s...@intel.com 
mailto:saisai.s...@intel.com wrote:


I think you can try to use ”spark.metrics.conf” to manually
specify the path of metrics.properties, but the prerequisite is
that each container should find this file in their local FS
because this file is loaded locally.

Besides I think this might be a kind of workaround, a better
solution is to fix this by some other solutions.

Thanks

Jerry

*From:*Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com
mailto:vladimir.tretya...@sematext.com]
*Sent:* Thursday, September 11, 2014 10:08 PM
*Cc:* user@spark.apache.org mailto:user@spark.apache.org
*Subject:* Re: JMXSink for YARN deployment

Hi Shao, thx for explanation, any ideas how to fix it? Where
should I put metrics.properties file?

On Thu, Sep 11, 2014 at 4:18 PM, Shao, Saisai
saisai.s...@intel.com mailto:saisai.s...@intel.com wrote:

Hi，

I’m guessing the problem is that driver or executor cannot get the
metrics.properties configuration file in the yarn container, so
metrics system cannot load the right sinks.

Thanks

Jerry

*From:*Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com
mailto:vladimir.tretya...@sematext.com]
*Sent:* Thursday, September 11, 2014 7:30 PM
*To:* user@spark.apache.org mailto:user@spark.apache.org
*Subject:* JMXSink for YARN deployment

Hello, we are in Sematext (https://apps.sematext.com/) are writing
Monitoring tool for Spark and we came across one question:

How to enable JMX metrics for YARN deployment?

We put *.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink

to file $SPARK_HOME/conf/metrics.properties but it doesn't work.

Everything works in Standalone mode, but not in YARN mode.

Can somebody help?

Thx!

PS: I've found also

https://stackoverflow.com/questions/23529404/spark-on-yarn-how-to-send-metrics-to-graphite-sink/25786112
without answer.

Re: JMXSink for YARN deployment

[Vladimir Tretyakov]

Hi, Kousuke,

Can you please explain a bit detailed what do you mean, I am new in Spark,
looked at https://spark.apache.org/docs/latest/submitting-applications.html
seems there is no '--files' option.

I just have to add '--files /path-to-metrics.properties' ? Undocumented
ability?

Thx for answer.



On Thu, Sep 11, 2014 at 5:55 PM, Kousuke Saruta saru...@oss.nttdata.co.jp
wrote:

  Hi Vladimir

 How about use --files option with spark-submit?

 - Kousuke


 (2014/09/11 23:43), Vladimir Tretyakov wrote:

  Hi again, yeah , I've tried to use ” spark.metrics.conf” before my
 question in ML, had no  luck:(
 Any other ideas from somebody?
 Seems nobody use metrics in YARN deployment mode.
 How about Mesos? I didn't try but maybe Spark has the same difficulties on
 Mesos?

  PS: Spark is great thing in general, will be nice to see metrics in
 YARN/Mesos mode, not only in Standalone:)


 On Thu, Sep 11, 2014 at 5:25 PM, Shao, Saisai saisai.s...@intel.com
 wrote:

  I think you can try to use ” spark.metrics.conf” to manually specify
 the path of metrics.properties, but the prerequisite is that each container
 should find this file in their local FS because this file is loaded locally.



 Besides I think this might be a kind of workaround, a better solution is
 to fix this by some other solutions.



 Thanks

 Jerry



 *From:* Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com]
 *Sent:* Thursday, September 11, 2014 10:08 PM
 *Cc:* user@spark.apache.org
 *Subject:* Re: JMXSink for YARN deployment



 Hi Shao, thx for explanation, any ideas how to fix it? Where should I put
 metrics.properties file?



 On Thu, Sep 11, 2014 at 4:18 PM, Shao, Saisai saisai.s...@intel.com
 wrote:

 Hi，



 I’m guessing the problem is that driver or executor cannot get the
 metrics.properties configuration file in the yarn container, so metrics
 system cannot load the right sinks.



 Thanks

 Jerry



 *From:* Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com]
 *Sent:* Thursday, September 11, 2014 7:30 PM
 *To:* user@spark.apache.org
 *Subject:* JMXSink for YARN deployment



 Hello, we are in Sematext (https://apps.sematext.com/) are writing
 Monitoring tool for Spark and we came across one question:



 How to enable JMX metrics for YARN deployment?



 We put *.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink

 to file $SPARK_HOME/conf/metrics.properties but it doesn't work.



 Everything works in Standalone mode, but not in YARN mode.



 Can somebody help?



 Thx!



 PS: I've found also
 https://stackoverflow.com/questions/23529404/spark-on-yarn-how-to-send-metrics-to-graphite-sink/25786112
 without answer.