Re: Profiling spark application
Give a look at this: https://github.com/LucaCanali/sparkMeasure On 2022/1/20 1:18, Prasad Bhalerao wrote: Is there any way we can profile spark applications which will show no. of invocations of spark api and their execution time etc etc just the way jprofiler shows all the details? - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: Profiling spark application
You can take a look at jvm profiler that was open sourced by uber: https://github.com/uber-common/jvm-profiler On Thu, Jan 20, 2022 at 11:20 AM Prasad Bhalerao < prasadbhalerao1...@gmail.com> wrote: > Hi, > > It will require code changes and I am looking at some third party code , I > am looking for something which I can just hook to jvm and get the stats.. > > Thanks, > Prasad > > On Thu, Jan 20, 2022 at 11:00 AM Sonal Goyal > wrote: > >> Hi Prasad, >> >> Have you checked the SparkListener - >> https://mallikarjuna_g.gitbooks.io/spark/content/spark-SparkListener.html >> ? >> >> Cheers, >> Sonal >> https://github.com/zinggAI/zingg >> >> >> >> On Thu, Jan 20, 2022 at 10:49 AM Prasad Bhalerao < >> prasadbhalerao1...@gmail.com> wrote: >> >>> Hello, >>> >>> Is there any way we can profile spark applications which will show no. >>> of invocations of spark api and their execution time etc etc just the way >>> jprofiler shows all the details? >>> >>> >>> Thanks, >>> Prasad >>> >> -- Thanks Deepak www.bigdatabig.com www.keosha.net
Re: Profiling spark application
Hi, It will require code changes and I am looking at some third party code , I am looking for something which I can just hook to jvm and get the stats.. Thanks, Prasad On Thu, Jan 20, 2022 at 11:00 AM Sonal Goyal wrote: > Hi Prasad, > > Have you checked the SparkListener - > https://mallikarjuna_g.gitbooks.io/spark/content/spark-SparkListener.html > ? > > Cheers, > Sonal > https://github.com/zinggAI/zingg > > > > On Thu, Jan 20, 2022 at 10:49 AM Prasad Bhalerao < > prasadbhalerao1...@gmail.com> wrote: > >> Hello, >> >> Is there any way we can profile spark applications which will show no. of >> invocations of spark api and their execution time etc etc just the way >> jprofiler shows all the details? >> >> >> Thanks, >> Prasad >> >
Re: Profiling spark application
Hi Prasad, Have you checked the SparkListener - https://mallikarjuna_g.gitbooks.io/spark/content/spark-SparkListener.html ? Cheers, Sonal https://github.com/zinggAI/zingg On Thu, Jan 20, 2022 at 10:49 AM Prasad Bhalerao < prasadbhalerao1...@gmail.com> wrote: > Hello, > > Is there any way we can profile spark applications which will show no. of > invocations of spark api and their execution time etc etc just the way > jprofiler shows all the details? > > > Thanks, > Prasad >
