Re: Query about how to estimate cpu usage for spark

2016-05-19 Thread Mich Talebzadeh
Please note taht CPU usage varies with time, it is not a fixed value First have a look at spark GUI that runs under port 4040 under tab jobs Then use jps to identify the spark process jps|grep SparkSubmit Using the process name start jmonitor on the OS and specify SparkSubmit process, It will

Query about how to estimate cpu usage for spark

2016-05-19 Thread Wang Jiaye
For MR job, there is job counter to provide CPU ms information while I cannot find a similar metrics in Spark which is quite useful. Do anyone know about this?