Hello. Found that there is also Spark metric Sink like MetricsServlet. which is enabled by default:
https://apache.googlesource.com/spark/+/refs/heads/master/core/src/main/scala/org/apache/spark/metrics/MetricsConfig.scala#40 Tried urls: On master: http://localhost:8080/metrics/master/json/ http://localhost:8080/metrics/applications/json On slaves (with workers): http://localhost:4040/metrics/json/ got information I need. Questions: 1. Will URLs for masted work in YARN (client/server mode) and Mesos modes? Or this is only for Standalone mode? 2. Will URL for slave also work for modes other than Standalone? Why are there 2 ways to get information, REST API and this Sink? Best regards, Vladimir. On Mon, Sep 12, 2016 at 3:53 PM, Vladimir Tretyakov < vladimir.tretya...@sematext.com> wrote: > Hello Saisai Shao, Jacek Laskowski , thx for information. > > We are working on Spark monitoring tool and our users have different setup > modes (Standalone, Mesos, YARN). > > Looked at code, found: > > /** > * Attempt to start a Jetty server bound to the supplied hostName:port using > the given > * context handlers. > * > * If the desired port number is contended, continues > *incrementing ports until a free port is** * found*. Return the jetty Server > object, the chosen port, and a mutable collection of handlers. > */ > > It seems most generic way (which will work for most users) will be start > looking at ports: > > spark.ui.port (4040 by default) > spark.ui.port + 1 > spark.ui.port + 2 > spark.ui.port + 3 > ... > > Until we will get responses from Spark. > > PS: yeah they may be some intersections with some other applications for > some setups, in this case we may ask users about these exceptions and do > our housework around them. > > Best regards, Vladimir. > > On Mon, Sep 12, 2016 at 12:07 PM, Saisai Shao <sai.sai.s...@gmail.com> > wrote: > >> Here is the yarn RM REST API for you to refer ( >> http://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop- >> yarn-site/ResourceManagerRest.html). You can use these APIs to query >> applications running on yarn. >> >> On Sun, Sep 11, 2016 at 11:25 PM, Jacek Laskowski <ja...@japila.pl> >> wrote: >> >>> Hi Vladimir, >>> >>> You'd have to talk to your cluster manager to query for all the >>> running Spark applications. I'm pretty sure YARN and Mesos can do that >>> but unsure about Spark Standalone. This is certainly not something a >>> Spark application's web UI could do for you since it is designed to >>> handle the single Spark application. >>> >>> Pozdrawiam, >>> Jacek Laskowski >>> ---- >>> https://medium.com/@jaceklaskowski/ >>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark >>> Follow me at https://twitter.com/jaceklaskowski >>> >>> >>> On Sun, Sep 11, 2016 at 11:18 AM, Vladimir Tretyakov >>> <vladimir.tretya...@sematext.com> wrote: >>> > Hello Jacek, thx a lot, it works. >>> > >>> > Is there a way how to get list of running applications from REST API? >>> Or I >>> > have to try connect 4040 4041... 40xx ports and check if ports answer >>> > something? >>> > >>> > Best regards, Vladimir. >>> > >>> > On Sat, Sep 10, 2016 at 6:00 AM, Jacek Laskowski <ja...@japila.pl> >>> wrote: >>> >> >>> >> Hi, >>> >> >>> >> That's correct. One app one web UI. Open 4041 and you'll see the other >>> >> app. >>> >> >>> >> Jacek >>> >> >>> >> >>> >> On 9 Sep 2016 11:53 a.m., "Vladimir Tretyakov" >>> >> <vladimir.tretya...@sematext.com> wrote: >>> >>> >>> >>> Hello again. >>> >>> >>> >>> I am trying to play with Spark version "2.11-2.0.0". >>> >>> >>> >>> Problem that REST API and UI shows me different things. >>> >>> >>> >>> I've stared 2 applications from "examples set": opened 2 consoles >>> and run >>> >>> following command in each: >>> >>> >>> >>> ./bin/spark-submit --class org.apache.spark.examples.SparkPi >>> --master >>> >>> spark://wawanawna:7077 --executor-memory 2G --total-executor-cores >>> 30 >>> >>> examples/jars/spark-examples_2.11-2.0.0.jar 10000 >>> >>> >>> >>> Request to API endpoint: >>> >>> >>> >>> http://localhost:4040/api/v1/applications >>> >>> >>> >>> returned me following JSON: >>> >>> >>> >>> [ { >>> >>> "id" : "app-20160909184529-0016", >>> >>> "name" : "Spark Pi", >>> >>> "attempts" : [ { >>> >>> "startTime" : "2016-09-09T15:45:25.047GMT", >>> >>> "endTime" : "1969-12-31T23:59:59.999GMT", >>> >>> "lastUpdated" : "2016-09-09T15:45:25.047GMT", >>> >>> "duration" : 0, >>> >>> "sparkUser" : "", >>> >>> "completed" : false, >>> >>> "startTimeEpoch" : 1473435925047, >>> >>> "endTimeEpoch" : -1, >>> >>> "lastUpdatedEpoch" : 1473435925047 >>> >>> } ] >>> >>> } ] >>> >>> >>> >>> so response contains information only about 1 application. >>> >>> >>> >>> But in reality I've started 2 applications and Spark UI shows me 2 >>> >>> RUNNING application (please see screenshot). >>> >>> >>> >>> Does anybody maybe know answer why API and UI shows different things? >>> >>> >>> >>> >>> >>> Best regards, Vladimir. >>> >>> >>> >>> >>> >>> On Tue, Aug 30, 2016 at 3:52 PM, Vijay Kiran <m...@vijaykiran.com> >>> wrote: >>> >>>> >>> >>>> Hi Otis, >>> >>>> >>> >>>> Did you check the REST API as documented in >>> >>>> http://spark.apache.org/docs/latest/monitoring.html >>> >>>> >>> >>>> Regards, >>> >>>> Vijay >>> >>>> >>> >>>> > On 30 Aug 2016, at 14:43, Otis Gospodnetić >>> >>>> > <otis.gospodne...@gmail.com> wrote: >>> >>>> > >>> >>>> > Hi Mich and Vijay, >>> >>>> > >>> >>>> > Thanks! I forgot to include an important bit - I'm looking for a >>> >>>> > programmatic way to get Spark metrics when running Spark under >>> YARN - so JMX >>> >>>> > or API of some kind. >>> >>>> > >>> >>>> > Thanks, >>> >>>> > Otis >>> >>>> > -- >>> >>>> > Monitoring - Log Management - Alerting - Anomaly Detection >>> >>>> > Solr & Elasticsearch Consulting Support Training - >>> >>>> > http://sematext.com/ >>> >>>> > >>> >>>> > >>> >>>> > On Tue, Aug 30, 2016 at 6:59 AM, Mich Talebzadeh >>> >>>> > <mich.talebza...@gmail.com> wrote: >>> >>>> > Spark UI regardless of deployment mode Standalone, yarn etc runs >>> on >>> >>>> > port 4040 by default that can be accessed directly >>> >>>> > >>> >>>> > Otherwise one can specify a specific port with --conf >>> >>>> > "spark.ui.port=55555" for example 55555 >>> >>>> > >>> >>>> > HTH >>> >>>> > >>> >>>> > Dr Mich Talebzadeh >>> >>>> > >>> >>>> > LinkedIn >>> >>>> > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJ >>> d6zP6AcPCCdOABUrV8Pw >>> >>>> > >>> >>>> > http://talebzadehmich.wordpress.com >>> >>>> > >>> >>>> > Disclaimer: Use it at your own risk. Any and all responsibility >>> for >>> >>>> > any loss, damage or destruction of data or any other property >>> which may >>> >>>> > arise from relying on this email's technical content is explicitly >>> >>>> > disclaimed. The author will in no case be liable for any monetary >>> damages >>> >>>> > arising from such loss, damage or destruction. >>> >>>> > >>> >>>> > >>> >>>> > On 30 August 2016 at 11:48, Vijay Kiran <m...@vijaykiran.com> >>> wrote: >>> >>>> > >>> >>>> > From Yarm RM UI, find the spark application Id, and in the >>> application >>> >>>> > details, you can click on the “Tracking URL” which should give >>> you the Spark >>> >>>> > UI. >>> >>>> > >>> >>>> > ./Vijay >>> >>>> > >>> >>>> > > On 30 Aug 2016, at 07:53, Otis Gospodnetić >>> >>>> > > <otis.gospodne...@gmail.com> wrote: >>> >>>> > > >>> >>>> > > Hi, >>> >>>> > > >>> >>>> > > When Spark is run on top of YARN, where/how can one get Spark >>> >>>> > > metrics? >>> >>>> > > >>> >>>> > > Thanks, >>> >>>> > > Otis >>> >>>> > > -- >>> >>>> > > Monitoring - Log Management - Alerting - Anomaly Detection >>> >>>> > > Solr & Elasticsearch Consulting Support Training - >>> >>>> > > http://sematext.com/ >>> >>>> > > >>> >>>> > >>> >>>> > >>> >>>> > ------------------------------------------------------------ >>> --------- >>> >>>> > To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>> >>>> > >>> >>>> > >>> >>>> > >>> >>>> >>> >>>> >>> >>>> ------------------------------------------------------------ >>> --------- >>> >>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>> >>>> >>> >>> >>> >>> >>> >>> >>> >>> ------------------------------------------------------------ >>> --------- >>> >>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>> > >>> > >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>> >>> >> >
