Hi Vladimir, You'd have to talk to your cluster manager to query for all the running Spark applications. I'm pretty sure YARN and Mesos can do that but unsure about Spark Standalone. This is certainly not something a Spark application's web UI could do for you since it is designed to handle the single Spark application.
Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Sun, Sep 11, 2016 at 11:18 AM, Vladimir Tretyakov <vladimir.tretya...@sematext.com> wrote: > Hello Jacek, thx a lot, it works. > > Is there a way how to get list of running applications from REST API? Or I > have to try connect 4040 4041... 40xx ports and check if ports answer > something? > > Best regards, Vladimir. > > On Sat, Sep 10, 2016 at 6:00 AM, Jacek Laskowski <ja...@japila.pl> wrote: >> >> Hi, >> >> That's correct. One app one web UI. Open 4041 and you'll see the other >> app. >> >> Jacek >> >> >> On 9 Sep 2016 11:53 a.m., "Vladimir Tretyakov" >> <vladimir.tretya...@sematext.com> wrote: >>> >>> Hello again. >>> >>> I am trying to play with Spark version "2.11-2.0.0". >>> >>> Problem that REST API and UI shows me different things. >>> >>> I've stared 2 applications from "examples set": opened 2 consoles and run >>> following command in each: >>> >>> ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master >>> spark://wawanawna:7077 --executor-memory 2G --total-executor-cores 30 >>> examples/jars/spark-examples_2.11-2.0.0.jar 10000 >>> >>> Request to API endpoint: >>> >>> http://localhost:4040/api/v1/applications >>> >>> returned me following JSON: >>> >>> [ { >>> "id" : "app-20160909184529-0016", >>> "name" : "Spark Pi", >>> "attempts" : [ { >>> "startTime" : "2016-09-09T15:45:25.047GMT", >>> "endTime" : "1969-12-31T23:59:59.999GMT", >>> "lastUpdated" : "2016-09-09T15:45:25.047GMT", >>> "duration" : 0, >>> "sparkUser" : "", >>> "completed" : false, >>> "startTimeEpoch" : 1473435925047, >>> "endTimeEpoch" : -1, >>> "lastUpdatedEpoch" : 1473435925047 >>> } ] >>> } ] >>> >>> so response contains information only about 1 application. >>> >>> But in reality I've started 2 applications and Spark UI shows me 2 >>> RUNNING application (please see screenshot). >>> >>> Does anybody maybe know answer why API and UI shows different things? >>> >>> >>> Best regards, Vladimir. >>> >>> >>> On Tue, Aug 30, 2016 at 3:52 PM, Vijay Kiran <m...@vijaykiran.com> wrote: >>>> >>>> Hi Otis, >>>> >>>> Did you check the REST API as documented in >>>> http://spark.apache.org/docs/latest/monitoring.html >>>> >>>> Regards, >>>> Vijay >>>> >>>> > On 30 Aug 2016, at 14:43, Otis Gospodnetić >>>> > <otis.gospodne...@gmail.com> wrote: >>>> > >>>> > Hi Mich and Vijay, >>>> > >>>> > Thanks! I forgot to include an important bit - I'm looking for a >>>> > programmatic way to get Spark metrics when running Spark under YARN - so >>>> > JMX >>>> > or API of some kind. >>>> > >>>> > Thanks, >>>> > Otis >>>> > -- >>>> > Monitoring - Log Management - Alerting - Anomaly Detection >>>> > Solr & Elasticsearch Consulting Support Training - >>>> > http://sematext.com/ >>>> > >>>> > >>>> > On Tue, Aug 30, 2016 at 6:59 AM, Mich Talebzadeh >>>> > <mich.talebza...@gmail.com> wrote: >>>> > Spark UI regardless of deployment mode Standalone, yarn etc runs on >>>> > port 4040 by default that can be accessed directly >>>> > >>>> > Otherwise one can specify a specific port with --conf >>>> > "spark.ui.port=55555" for example 55555 >>>> > >>>> > HTH >>>> > >>>> > Dr Mich Talebzadeh >>>> > >>>> > LinkedIn >>>> > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>> > >>>> > http://talebzadehmich.wordpress.com >>>> > >>>> > Disclaimer: Use it at your own risk. Any and all responsibility for >>>> > any loss, damage or destruction of data or any other property which may >>>> > arise from relying on this email's technical content is explicitly >>>> > disclaimed. The author will in no case be liable for any monetary damages >>>> > arising from such loss, damage or destruction. >>>> > >>>> > >>>> > On 30 August 2016 at 11:48, Vijay Kiran <m...@vijaykiran.com> wrote: >>>> > >>>> > From Yarm RM UI, find the spark application Id, and in the application >>>> > details, you can click on the “Tracking URL” which should give you the >>>> > Spark >>>> > UI. >>>> > >>>> > ./Vijay >>>> > >>>> > > On 30 Aug 2016, at 07:53, Otis Gospodnetić >>>> > > <otis.gospodne...@gmail.com> wrote: >>>> > > >>>> > > Hi, >>>> > > >>>> > > When Spark is run on top of YARN, where/how can one get Spark >>>> > > metrics? >>>> > > >>>> > > Thanks, >>>> > > Otis >>>> > > -- >>>> > > Monitoring - Log Management - Alerting - Anomaly Detection >>>> > > Solr & Elasticsearch Consulting Support Training - >>>> > > http://sematext.com/ >>>> > > >>>> > >>>> > >>>> > --------------------------------------------------------------------- >>>> > To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>>> > >>>> > >>>> > >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>>> >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
