Thanks for the response Mayur. I was seeing the webui of 0.9.0 spark. I see lots of detailed statistics in the newer 1.0.0-snapshot version. The only thing I found missing was the actual code that I had typed in at the spark-shell prompt, but I can always get it from the shell history.
On 26-Mar-2014, at 7:53 am, Mayur Rustagi <mayur.rust...@gmail.com> wrote: > Time taken is shown in Shark shell web ui (hosted on 4040 port). > Also memory used is shown in terms of Storage of RDD, how much shuffle data > was written & read during the process is also highlighted thr. > > Mayur Rustagi > Ph: +1 (760) 203 3257 > http://www.sigmoidanalytics.com > @mayur_rustagi > > > > On Tue, Mar 25, 2014 at 6:04 AM, Bharath Bhushan <manku.ti...@outlook.com> > wrote: > Is there a way to see the resource usage of each spark-shell command — say > time taken and memory used? > I checked the WebUI of spark-shell and of the master and I don’t see any such > breakdown. I see the time taken in the INFO logs but nothing about memory > usage. It would also be nice to track the time taken in the spark-shell web > UI. > > — > Thanks >
