Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

t;>>> rohitk >>>>>> >>>>>> On Sun, Mar 25, 2018 at 1:25 PM, Shmuel Blitz < >>>>>> shmuel.bl...@similarweb.com> wrote: >>>>>> >>>>>&g

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

t; job? >>>>>>> >>>>>>> Did you configure it on the spark-default.conf? >>>>>>> >>>>>>> On Sun, 25 Mar 2018 at 10:34 Shmuel Blitz < >>>>>>> shmuel.bl...@similarweb.com> wrote: &

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

r i >>>>>>>> should use spark.jars option? anyone who could give an example how it >>>>>>>> should be, and if i the path for the jar should be an hdfs path as i'm >>>>>>>> using it in cluster mode. >>>>>>>&

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

>>> shmuel.bl...@similarweb.com> wrote: >>>>>>>> >>>>>>>>> Hi Rohit, >>>>>>>>> >>>>>>>>> Thanks for sharing this great tool. >>>>>>>>> I trie

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

t; On Thu, Mar 22, 2018 at 4:22 PM, Rohit Karlupia <roh...@qubole.com >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> Thanks everyone! >>>>>>>>>> Please share how it works

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

t;>>>>> >>>>>>>>> thanks, >>>>>>>>> rohitk >>>>>>>>> >>>>>>>>> >>>>>>>>> >

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

t; >>>>>>>> >>>>>>>> On Thu, Mar 22, 2018 at 10:18 AM, Fawze Abujaber <fawz...@gmail.com >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> It's super amazing i see it was tested on sp

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

I ran it on a single job. SparkLens has an overhead on the job duration. I'm not ready to enable it by default on all our jobs. Attached is the output. Still trying to understand what exactly it means. On Sun, Mar 25, 2018 at 10:40 AM, Fawze Abujaber wrote: > Nice! > >

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

s still part of Cloudera's main >>>>>>> versions? >>>>>>> >>>>>>> We have a vast Spark applications with version 1.6.0 >>>>>>> >>>>>>> On Thu, Mar 22, 2018 at 6:38 AM, Holden Karau <ho

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

Just to let you know, I have managed to run SparkLens on our cluster. I switched to the spark_1.6 branch, and also compiled against the specific image of Spark we are using (cdh5.7.6). Now I need to figure out what the output means... :P Shmuel On Fri, Mar 23, 2018 at 7:24 PM, Fawze Abujaber

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

Quick question: how to add the --jars /path/to/sparklens_2.11-0.1.0.jar to the spark-default conf, should it be using: spark.driver.extraClassPath /path/to/sparklens_2.11-0.1.0.jar or i should use spark.jars option? anyone who could give an example how it should be, and if i the path for the

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

Hi Shmuel, Did you compile the code against the right branch for Spark 1.6. I tested it and it looks working and now i'm testing the branch for a wide tests, Please use the branch for Spark 1.6 On Fri, Mar 23, 2018 at 12:43 AM, Shmuel Blitz wrote: > Hi Rohit, > >

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

Hi Rohit, Thanks for sharing this great tool. I tried running a spark job with the tool, but it failed with an *IncompatibleClassChangeError *Exception. I have opened an issue on Github.( https://github.com/qubole/sparklens/issues/1) Shmuel On Thu, Mar 22, 2018 at 5:05 PM, Shmuel Blitz

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

Thanks. We will give this a try and report back. Shmuel On Thu, Mar 22, 2018 at 4:22 PM, Rohit Karlupia wrote: > Thanks everyone! > Please share how it works and how it doesn't. Both help. > > Fawaze, just made few changes to make this work with spark 1.6. Can you > please

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

Thanks everyone! Please share how it works and how it doesn't. Both help. Fawaze, just made few changes to make this work with spark 1.6. Can you please try building from branch *spark_1.6* thanks, rohitk On Thu, Mar 22, 2018 at 10:18 AM, Fawze Abujaber wrote: > It's

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

be a useful guide on the path towards tuning applications for lower >>> runtime or cost. >>> >>> Please clone from here: https://github.com/qubole/sparklens >>> Old blogpost: https://www.qubole.com/blog/introducing-quboles- >>> spark-tuning-tool/

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

cations for lower >> runtime or cost. >> >> Please clone from here: https://github.com/qubole/sparklens >> Old blogpost: >> https://www.qubole.com/blog/introducing-quboles-spark-tuning-tool/ >> >> thanks, >> rohitk >> >> PS: Thanks for the patience. It took couple of months to get back on >> this. >> >> >> >> >> > -- Twitter: https://twitter.com/holdenkarau

Re: Open sourcing Sparklens: Qubole's Spark Tuning Tool

ications and > can be a useful guide on the path towards tuning applications for lower > runtime or cost. > > Please clone from here: https://github.com/qubole/sparklens > Old blogpost: https://www.qubole.com/blog/introducing-quboles- > spark-tuning-tool/ > > thanks, > ro

Open sourcing Sparklens: Qubole's Spark Tuning Tool

/sparklens Old blogpost: https://www.qubole.com/blog/introducing-quboles-spark- tuning-tool/ thanks, rohitk PS: Thanks for the patience. It took couple of months to get back on this.

Re: Spark Tuning Tool

> On Mon, Jan 22, 2018 at 9:01 PM, Rohit Karlupia <roh...@qubole.com> > wrote: > >> > >> Hi, > >> > >> I have been working on making the performance tuning of spark > applications > >> bit easier. We have just released the

Re: Spark Tuning Tool

.com> wrote: >> >> Hi, >> >> I have been working on making the performance tuning of spark applications >> bit easier. We have just released the beta version of the tool on Qubole. >> >> https://www.qubole.com/blog/introducing-quboles-spark-tuning-tool/ &

Re: Spark Tuning Tool

have just released the beta version of the tool on Qubole. > > https://www.qubole.com/blog/introducing-quboles-spark-tuning-tool/ > > This is not OSS yet but we would like to contribute it to OSS. Fishing > for some interest in the community if people find this work interesting

Re: Spark Tuning Tool

ave just released the beta version of the >> tool on Qubole. >> >> https://www.qubole.com/blog/introducing-quboles-spark-tuning-tool/ >> >> This is not OSS yet but we would like to contribute it to OSS. Fishing >> for some interest in the community if people find this work interesting and >> would like to try to it out. >> >> thanks, >> Rohit Karlupia >> >> >> > > > -- > >

Re: Spark Tuning Tool

lications > bit easier. We have just released the beta version of the tool on Qubole. > > https://www.qubole.com/blog/introducing-quboles-spark-tuning-tool/ > > This is not OSS yet but we would like to contribute it to OSS. Fishing > for some interest in the community if

Re: Spark Tuning Tool

nterested. >>> >>> On 23 Jan. 2018 4:01 pm, "Rohit Karlupia" <roh...@qubole.com> wrote: >>> >>>> Hi, >>>> >>>> I have been working on making the performance tuning of spark >>>> applications bit easier. W

Re: Spark Tuning Tool

; On 23 Jan. 2018 4:01 pm, "Rohit Karlupia" <roh...@qubole.com> wrote: >> >>> Hi, >>> >>> I have been working on making the performance tuning of spark >>> applications bit easier. We have just released the beta version of the >>> tool on Qu

Re: Spark Tuning Tool

t; wrote: > >> Hi, >> >> I have been working on making the performance tuning of spark >> applications bit easier. We have just released the beta version of the >> tool on Qubole. >> >> https://www.qubole.com/blog/introducing-quboles-spark-tuning-tool/ &

Re: Spark Tuning Tool

; > https://www.qubole.com/blog/introducing-quboles-spark-tuning-tool/ > > This is not OSS yet but we would like to contribute it to OSS. Fishing > for some interest in the community if people find this work interesting and > would like to try to it out. > > thanks, > Rohit Karlupia > > >

Spark Tuning Tool

Hi, I have been working on making the performance tuning of spark applications bit easier. We have just released the beta version of the tool on Qubole. https://www.qubole.com/blog/introducing-quboles-spark-tuning-tool/ This is not OSS yet but we would like to contribute it to OSS. Fishing