Hi, I'm not surprised to see Hadoop jars on the driver (yet I couldn't explain exactly why they need to be there). I can't find a way now to display the classpath for executors.
Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Tue, Jun 7, 2016 at 10:05 PM, satish saley <satishsale...@gmail.com> wrote: > Thank you Jacek. > In case of YARN, I see that hadoop jars are present in system classpath for > Driver. Will it be the same for all executors? > > On Tue, Jun 7, 2016 at 11:22 AM, Jacek Laskowski <ja...@japila.pl> wrote: >> >> Ouch, I made a mistake :( Sorry. >> >> You've asked about spark **history** server. It's pretty much the same. >> >> HistoryServer is a web interface for completed and running (aka >> incomplete) Spark applications. It uses EventLoggingListener to >> collect events as JSON using org.apache.spark.util.JsonProtocol >> object. >> >> Pozdrawiam, >> Jacek Laskowski >> ---- >> https://medium.com/@jaceklaskowski/ >> Mastering Apache Spark http://bit.ly/mastering-apache-spark >> Follow me at https://twitter.com/jaceklaskowski >> >> >> On Tue, Jun 7, 2016 at 8:18 PM, Jacek Laskowski <ja...@japila.pl> wrote: >> > Hi, >> > >> > It is the driver - see the port. Is this 4040 or similar? It's started >> > when SparkContext starts and is controlled by spark.ui.enabled. >> > >> > spark.ui.enabled (default: true) = controls whether the web UI is >> > started or not. >> > >> > It's through JobProgressListener which is the SparkListener for web UI >> > that the console knows what happens under the covers (and can >> > calculate the stats). >> > >> > BTW, spark.ui.port (default: 4040) controls the port Web UI binds to. >> > >> > Pozdrawiam, >> > Jacek Laskowski >> > ---- >> > https://medium.com/@jaceklaskowski/ >> > Mastering Apache Spark http://bit.ly/mastering-apache-spark >> > Follow me at https://twitter.com/jaceklaskowski >> > >> > >> > On Tue, Jun 7, 2016 at 8:11 PM, satish saley <satishsale...@gmail.com> >> > wrote: >> >> Hi, >> >> In spark history server, we see environment tab. Is it show environment >> >> of >> >> Driver or Executor or both? >> >> >> >> Jobs >> >> Stages >> >> Storage >> >> Environment >> >> Executors >> >> > > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
