If you are profiling in standalone mode, I recommend you to try with Java
Mission Control.
You just need to start app with these params:
-XX:+UnlockCommercialFeatures -XX:+FlightRecorder
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=$YOUR_PORT
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
And them connect with profiling agent, and perform recording. I have strong
opinion, that you have epoll wait time spent in one concrete thread, but
not in all spawned threads.
On Tue, Apr 5, 2016 at 1:34 PM, Dmitry Olshansky
wrote:
> Hi list,
>
> I'm curious as to what are the best practices of profiling spark apps? So
> far I tried following this guide with hprof and/or yourkit but the profile
> looks strange:
>
> https://cwiki.apache.org/confluence/display/SPARK/Profiling+Spark+Applications+Using+YourKit
>
> 55% of time spent in EPollWait. However I'm using standalone mode with
> local master without starting separate daemon (could it be that I should?)
>
> ---
> Dmitry Olshansky
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>