The application web ui is pretty useful. We have been adding more and more information to the web ui for easier performance analysis.
Look at Patrick Wendell's two talks at the Spark Summit for more information: http://spark-summit.org/summit-2013/ On Sat, Dec 28, 2013 at 8:12 PM, Hao Lin <[email protected]> wrote: > Hi folks, > > I am trying to test the performance on a couple of my Spark applications. > For benchmarking purpose, I am wondering if there is a good performance > analysis practice. The best way I can think of is to instrument log prints > and analyze the timestamps in logs on each node. > > The major metrics I am interested in are computation ratios (computation > time, data transferring time, basically a timeline of detailed events), > memory usage, disk throughput. Could I have some suggestions on how Spark > is benchmarked. > > Thanks, > > Max >
