Hi Sutanu , When you run your spark shell you would see below lines in your console
16/02/18 21:43:53 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4041 16/02/18 21:43:53 INFO Utils: Successfully started service 'SparkUI' on port 4041. 16/02/18 21:43:54 INFO SparkUI: Started SparkUI at http://xx.xx.xx.xxx:4041 As In my case instead of default port the UI started at 4041 port . Hope this helps. Thanks, Divya On 19 February 2016 at 07:09, Mich Talebzadeh <m...@peridale.co.uk> wrote: > Is 4040 port used in your host? It should be default > > > > Example > > > > *netstat -plten|grep 4040* > > > > tcp 0 0 :::4040 > :::* LISTEN 1009 42748209 *22778*/java > > > > *ps -ef|grep 22778* > > > > hduser 22778 22770 0 08:34 pts/1 00:01:18 /usr/java/latest/bin/java > -cp > /home/hduser/jars/jconn4.jar:/home/hduser/jars/ojdbc6.jar:/usr/lib/spark-1.5.2-bin-hadoop2.6/conf/:/usr/lib/spark-1.5.2-bin-hadoop2.6/lib/spark-assembly-1.5.2-hadoop2.6.0.jar:/usr/lib/spark-1.5.2-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/usr/lib/spark-1.5.2-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/usr/lib/spark-1.5.2-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/home/hduser/hadoop-2.6.0/etc/hadoop/ > -Dscala.usejavacp=true -Xms1G -Xmx1G -XX:MaxPermSize=256m > org.apache.spark.deploy.SparkSubmit --master spark://50.140.197.217:7077 > --class org.apache.spark.repl.Main --name Spark shell spark-shell > > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > > NOTE: The information in this email is proprietary and confidential. This > message is for the designated recipient only, if you are not the intended > recipient, you should destroy it immediately. Any information in this > message shall not be understood as given or endorsed by Peridale Technology > Ltd, its subsidiaries or their employees, unless expressly so stated. It is > the responsibility of the recipient to ensure that this email is virus > free, therefore neither Peridale Technology Ltd, its subsidiaries nor their > employees accept any responsibility. > > > > > > *From:* Sutanu Das [mailto:sd2...@att.com] > *Sent:* 18 February 2016 22:58 > *To:* Mich Talebzadeh <m...@peridale.co.uk>; user@spark.apache.org > > *Subject:* RE: Spark History Server NOT showing Jobs with Hortonworks > > > > Hi Mich, Community - Do I need to specify it in the properties file in my > spark-submit ? > > > > *From:* Mich Talebzadeh [mailto:m...@peridale.co.uk <m...@peridale.co.uk>] > > *Sent:* Thursday, February 18, 2016 4:28 PM > *To:* Sutanu Das; user@spark.apache.org > *Subject:* RE: Spark History Server NOT showing Jobs with Hortonworks > > > > The jobs are normally shown under <HOSTNAME>:4040/jobs/ in a normal set up > not using any vendor’s flavoiur > > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > > NOTE: The information in this email is proprietary and confidential. This > message is for the designated recipient only, if you are not the intended > recipient, you should destroy it immediately. Any information in this > message shall not be understood as given or endorsed by Peridale Technology > Ltd, its subsidiaries or their employees, unless expressly so stated. It is > the responsibility of the recipient to ensure that this email is virus > free, therefore neither Peridale Technology Ltd, its subsidiaries nor their > employees accept any responsibility. > > > > > > *From:* Sutanu Das [mailto:sd2...@att.com <sd2...@att.com>] > *Sent:* 18 February 2016 22:22 > *To:* user@spark.apache.org > *Subject:* Spark History Server NOT showing Jobs with Hortonworks > > > > Hi Community, > > > > Challenged with Spark issues with *Hortonworks* (HDP 2.3.2_Spark 1.4.1) > – The Spark History Server is NOT showing the Spark Running Jobs in Local > Mode > > > > The local-host:4040/app/v1 is ALSO not working > > > > How can I look at my local Spark job? > > > > > > # Generated by Apache Ambari. Fri Feb 5 00:37:06 2016 > > > > spark.history.kerberos.keytab none > > spark.history.kerberos.principal none > > spark.history.provider > org.apache.spark.deploy.yarn.history.YarnHistoryProvider > > spark.history.ui.port 18080 > > spark.yarn.containerLauncherMaxThreads 25 > > spark.yarn.driver.memoryOverhead 2048 > > spark.yarn.executor.memoryOverhead 2048 > > spark.yarn.historyServer.address has-dal-0001.corp.wayport.net:18080 > > spark.yarn.max.executor.failures 3 > > spark.yarn.preserve.staging.files false > > spark.yarn.queue default > > spark.yarn.scheduler.heartbeat.interval-ms 5000 > > spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService > > spark.yarn.submit.file.replication 3 > > > > *History Server * > > - *Timeline Service Location:* > http://has-dal-0002.corp.wayport.net:8188/ > - *Last Updated:* Feb 18, 2016 10:09:12 PM UTC > - *Service Started:* Feb 5, 2016 12:37:15 AM UTC > - *Current Time:* Feb 18, 2016 10:10:46 PM UTC > - *Timeline Service:* Timeline service is enabled > - *History Provider:* Apache Hadoop YARN Timeline Service > > >
