Hi folks, I am trying to test the performance on a couple of my Spark applications. For benchmarking purpose, I am wondering if there is a good performance analysis practice. The best way I can think of is to instrument log prints and analyze the timestamps in logs on each node.
The major metrics I am interested in are computation ratios (computation time, data transferring time, basically a timeline of detailed events), memory usage, disk throughput. Could I have some suggestions on how Spark is benchmarked. Thanks, Max
