Thanks!
*Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)* *Mobile : (213)-572-7858* *Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254 <http://www.linkedin.com/pub/parag-chaudhari/28/a55/254>* On Tue, Feb 28, 2017 at 12:53 PM, Shixiong(Ryan) Zhu < shixi...@databricks.com> wrote: > The REST APIs are not just for Spark history server. When an application > is running, you can use the REST APIs to talk to Spark UI HTTP server as > well. > > On Tue, Feb 28, 2017 at 10:46 AM, Parag Chaudhari <paragp...@gmail.com> > wrote: > >> ping... >> >> >> >> *Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)* >> *Mobile : (213)-572-7858 <(213)%20572-7858>* >> *Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254 >> <http://www.linkedin.com/pub/parag-chaudhari/28/a55/254>* >> >> >> On Wed, Feb 22, 2017 at 7:54 PM, Parag Chaudhari <paragp...@gmail.com> >> wrote: >> >>> Thanks! >>> >>> If spark does not log these events in event log then why spark history >>> server provides an API to get RDD information? >>> >>> From the documentation, >>> >>> /applications/[app-id]/storage/rdd A list of stored RDDs for the >>> given application. >>> >>> /applications/[app-id]/storage/rdd/[rdd-id] Details for the storage >>> status of a given RDD. >>> >>> >>> >>> >>> *Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)* >>> *Mobile : (213)-572-7858 <(213)%20572-7858>* >>> *Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254 >>> <http://www.linkedin.com/pub/parag-chaudhari/28/a55/254>* >>> >>> >>> On Wed, Feb 22, 2017 at 7:44 PM, Saisai Shao <sai.sai.s...@gmail.com> >>> wrote: >>> >>>> It is too verbose, and will significantly increase the size event log. >>>> >>>> Here is the comment in the code: >>>> >>>> // No-op because logging every update would be overkill >>>>> override def onBlockUpdated(event: SparkListenerBlockUpdated): Unit = {} >>>>> >>>>> >>>> On Thu, Feb 23, 2017 at 11:42 AM, Parag Chaudhari <paragp...@gmail.com> >>>> wrote: >>>> >>>>> Thanks a lot the information! >>>>> >>>>> Is there any reason why EventLoggingListener ignore this event? >>>>> >>>>> *Thanks,* >>>>> >>>>> >>>>> *Parag* >>>>> >>>>> On Wed, Feb 22, 2017 at 7:11 PM, Saisai Shao <sai.sai.s...@gmail.com> >>>>> wrote: >>>>> >>>>>> AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it >>>>>> will not be written into event-log, I think that's why you cannot get >>>>>> such >>>>>> info in history server. >>>>>> >>>>>> On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari <paragp...@gmail.com >>>>>> > wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am running spark shell in spark version 2.0.2. Here is my program, >>>>>>> >>>>>>> var myrdd = sc.parallelize(Array.range(1, 10)) >>>>>>> myrdd.setName("test") >>>>>>> myrdd.cache >>>>>>> myrdd.collect >>>>>>> >>>>>>> But I am not able to see any RDD info in "storage" tab in spark >>>>>>> history server. >>>>>>> >>>>>>> I looked at this >>>>>>> <https://forums.databricks.com/questions/117/why-is-my-rdd-not-showing-up-in-the-storage-tab-of.html> >>>>>>> but it is not helping as I have exact similar program mentioned there. >>>>>>> Can >>>>>>> anyone help? >>>>>>> >>>>>>> >>>>>>> *Thanks,* >>>>>>> >>>>>>> *Parag* >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >