Re: Spark metrics when running with YARN?

Hi, When I start Spark v1.6 (cdh5.8.0) in YARN Master mode I don't see API ( http://localhost:4040/api/v1/applications is unavailable) on port 4040. I started Spark application like this: spark-submit --master yarn-cluster --class org.apache.spark.examples.SparkPi /usr/lib/spark/examples

Re: Spark metrics when running with YARN?

Hello Saisai Shao. Thx for reminder, I know which component in which mode Spark has. But Mich Talebzadeh has written above that URL 4040 will work regardless mode user use, that's why I hoped it will be also true for metrics URL (since they are on the same port). I think you are right, better st

Re: Spark metrics when running with YARN?

H Vladimir, I think you mixed cluster manager and Spark application running on it, the master and workers are two components for Standalone cluster manager, the yarn counterparts are RM and NM. the URL you listed above is only worked for standalone master and workers. It would be more clear if yo

Re: Spark metrics when running with YARN?

Hello. Found that there is also Spark metric Sink like MetricsServlet. which is enabled by default: https://apache.googlesource.com/spark/+/refs/heads/master/core/src/main/scala/org/apache/spark/metrics/MetricsConfig.scala#40 Tried urls: On master: http://localhost:8080/metrics/master/json/ htt

Re: Spark metrics when running with YARN?

Hello Saisai Shao, Jacek Laskowski , thx for information. We are working on Spark monitoring tool and our users have different setup modes (Standalone, Mesos, YARN). Looked at code, found: /** * Attempt to start a Jetty server bound to the supplied hostName:port using the given * context handl

Re: Spark metrics when running with YARN?

Here is the yarn RM REST API for you to refer ( http://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html). You can use these APIs to query applications running on yarn. On Sun, Sep 11, 2016 at 11:25 PM, Jacek Laskowski wrote: > Hi Vladimir, > > You'd have to tal

Re: Spark metrics when running with YARN?

Hi Vladimir, You'd have to talk to your cluster manager to query for all the running Spark applications. I'm pretty sure YARN and Mesos can do that but unsure about Spark Standalone. This is certainly not something a Spark application's web UI could do for you since it is designed to handle the si

Re: Spark metrics when running with YARN?

Hello Jacek, thx a lot, it works. Is there a way how to get list of running applications from REST API? Or I have to try connect 4040 4041... 40xx ports and check if ports answer something? Best regards, Vladimir. On Sat, Sep 10, 2016 at 6:00 AM, Jacek Laskowski wrote: > Hi, > > That's correct

Re: Spark metrics when running with YARN?

Hi, That's correct. One app one web UI. Open 4041 and you'll see the other app. Jacek On 9 Sep 2016 11:53 a.m., "Vladimir Tretyakov" < vladimir.tretya...@sematext.com> wrote: > Hello again. > > I am trying to play with Spark version "2.11-2.0.0". > > Problem that REST API and UI shows me differ

Re: Spark metrics when running with YARN?

Hi Otis, Did you check the REST API as documented in http://spark.apache.org/docs/latest/monitoring.html Regards, Vijay > On 30 Aug 2016, at 14:43, Otis Gospodnetić wrote: > > Hi Mich and Vijay, > > Thanks! I forgot to include an important bit - I'm looking for a > programmatic way to ge

Re: Spark metrics when running with YARN?

Hi Mich and Vijay, Thanks! I forgot to include an important bit - I'm looking for a *programmatic* way to get Spark metrics when running Spark under YARN - so JMX or API of some kind. Thanks, Otis -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Suppo

Re: Spark metrics when running with YARN?

Spark UI regardless of deployment mode Standalone, yarn etc runs on port 4040 by default that can be accessed directly Otherwise one can specify a specific port with --conf "spark.ui.port=5" for example 5 HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEA

Re: Spark metrics when running with YARN?

From Yarm RM UI, find the spark application Id, and in the application details, you can click on the “Tracking URL” which should give you the Spark UI. ./Vijay > On 30 Aug 2016, at 07:53, Otis Gospodnetić wrote: > > Hi, > > When Spark is run on top of YARN, where/how can one get Spark metric

Re: Spark metrics when running with YARN?

Have you checked spark UI on port HOST:4040 by default? HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com

Spark metrics when running with YARN?

Hi, When Spark is run on top of YARN, where/how can one get Spark metrics? Thanks, Otis -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/