Time taken is shown in Shark shell web ui (hosted on 4040 port). Also memory used is shown in terms of Storage of RDD, how much shuffle data was written & read during the process is also highlighted thr.
Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi <https://twitter.com/mayur_rustagi> On Tue, Mar 25, 2014 at 6:04 AM, Bharath Bhushan <manku.ti...@outlook.com>wrote: > Is there a way to see the resource usage of each spark-shell command — say > time taken and memory used? > I checked the WebUI of spark-shell and of the master and I don’t see any > such breakdown. I see the time taken in the INFO logs but nothing about > memory usage. It would also be nice to track the time taken in the > spark-shell web UI. > > — > Thanks
