Quick question: how to add the --jars /path/to/sparklens_2.11-0.1.0.jar to the spark-default conf, should it be using:
spark.driver.extraClassPath /path/to/sparklens_2.11-0.1.0.jar or i should use spark.jars option? anyone who could give an example how it should be, and if i the path for the jar should be an hdfs path as i'm using it in cluster mode. On Fri, Mar 23, 2018 at 6:33 AM, Fawze Abujaber <fawz...@gmail.com> wrote: > Hi Shmuel, > > Did you compile the code against the right branch for Spark 1.6. > > I tested it and it looks working and now i'm testing the branch for a wide > tests, Please use the branch for Spark 1.6 > > On Fri, Mar 23, 2018 at 12:43 AM, Shmuel Blitz < > shmuel.bl...@similarweb.com> wrote: > >> Hi Rohit, >> >> Thanks for sharing this great tool. >> I tried running a spark job with the tool, but it failed with an >> *IncompatibleClassChangeError >> *Exception. >> >> I have opened an issue on Github.(https://github.com/qub >> ole/sparklens/issues/1) >> >> Shmuel >> >> On Thu, Mar 22, 2018 at 5:05 PM, Shmuel Blitz < >> shmuel.bl...@similarweb.com> wrote: >> >>> Thanks. >>> >>> We will give this a try and report back. >>> >>> Shmuel >>> >>> On Thu, Mar 22, 2018 at 4:22 PM, Rohit Karlupia <roh...@qubole.com> >>> wrote: >>> >>>> Thanks everyone! >>>> Please share how it works and how it doesn't. Both help. >>>> >>>> Fawaze, just made few changes to make this work with spark 1.6. Can you >>>> please try building from branch *spark_1.6* >>>> >>>> thanks, >>>> rohitk >>>> >>>> >>>> >>>> On Thu, Mar 22, 2018 at 10:18 AM, Fawze Abujaber <fawz...@gmail.com> >>>> wrote: >>>> >>>>> It's super amazing .... i see it was tested on spark 2.0.0 and above, >>>>> what about Spark 1.6 which is still part of Cloudera's main versions? >>>>> >>>>> We have a vast Spark applications with version 1.6.0 >>>>> >>>>> On Thu, Mar 22, 2018 at 6:38 AM, Holden Karau <hol...@pigscanfly.ca> >>>>> wrote: >>>>> >>>>>> Super exciting! I look forward to digging through it this weekend. >>>>>> >>>>>> On Wed, Mar 21, 2018 at 9:33 PM ☼ R Nair (रविशंकर नायर) < >>>>>> ravishankar.n...@gmail.com> wrote: >>>>>> >>>>>>> Excellent. You filled a missing link. >>>>>>> >>>>>>> Best, >>>>>>> Passion >>>>>>> >>>>>>> On Wed, Mar 21, 2018 at 11:36 PM, Rohit Karlupia <roh...@qubole.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Happy to announce the availability of Sparklens as open source >>>>>>>> project. It helps in understanding the scalability limits of spark >>>>>>>> applications and can be a useful guide on the path towards tuning >>>>>>>> applications for lower runtime or cost. >>>>>>>> >>>>>>>> Please clone from here: https://github.com/qubole/sparklens >>>>>>>> Old blogpost: https://www.qubole.com/blog/introducing-quboles-sp >>>>>>>> ark-tuning-tool/ >>>>>>>> >>>>>>>> thanks, >>>>>>>> rohitk >>>>>>>> >>>>>>>> PS: Thanks for the patience. It took couple of months to get back >>>>>>>> on this. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>> Twitter: https://twitter.com/holdenkarau >>>>>> >>>>> >>>>> >>>> >>> >>> >>> -- >>> Shmuel Blitz >>> Big Data Developer >>> Email: shmuel.bl...@similarweb.com >>> www.similarweb.com >>> <https://www.facebook.com/SimilarWeb/> >>> <https://www.linkedin.com/company/429838/> >>> <https://twitter.com/similarweb> >>> >> >> >> >> -- >> Shmuel Blitz >> Big Data Developer >> Email: shmuel.bl...@similarweb.com >> www.similarweb.com >> <https://www.facebook.com/SimilarWeb/> >> <https://www.linkedin.com/company/429838/> >> <https://twitter.com/similarweb> >> > >
