;>>> spark.sql.shuffle.partitions. Alternatively you could try
>>>>>> increasing the value of spark.sql.shuffle.partitions to latest 750.
>>>>>>
>>>>>> thanks,
>>>>>> rohitk
>>>>>>
>>>>>> On Su
;>>
>>>>>>> On Sun, 25 Mar 2018 at 10:34 Shmuel Blitz <
>>>>>>> shmuel.bl...@similarweb.com> wrote:
>>>>>>>
>>>>>>>> Just to let you know, I have managed to run SparkLens on our
>>>
gt;>>> spark-default conf, should it be using:
>>>>>>>>
>>>>>>>> spark.driver.extraClassPath /path/to/sparklens_2.11-0.1.0.jar or i
>>>>>>>> should use spark.jars option? anyone who could give an example how it
t;>>>>>> Thanks for sharing this great tool.
>>>>>>>>> I tried running a spark job with the tool, but it failed with an
>>>>>>>>> *IncompatibleClassChangeError
>>>>>>>>> *Exception.
>>>>>
gt;>>> Thanks everyone!
>>>>>>>>>> Please share how it works and how it doesn't. Both help.
>>>>>>>>>>
>>>>>>>>>> Fawaze, just made few changes to make this work with spark 1.6.
>>>>>
h spark 1.6.
>>>>>>>>> Can you please try building from branch *spark_1.6*
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>> rohitk
>>>>>>>>>
>>>>>>>>>
&
;
>>>>>>>> thanks,
>>>>>>>> rohitk
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Mar 22, 2018 at 10:18 AM, Fawze Abujaber >>>>>&g
I ran it on a single job.
SparkLens has an overhead on the job duration. I'm not ready to enable it
by default on all our jobs.
Attached is the output.
Still trying to understand what exactly it means.
On Sun, Mar 25, 2018 at 10:40 AM, Fawze Abujaber wrote:
> Nice!
>
> Shmuel, Were you able to
>>>> We have a vast Spark applications with version 1.6.0
>>>>>>>
>>>>>>> On Thu, Mar 22, 2018 at 6:38 AM, Holden Karau
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Super exciting!
Just to let you know, I have managed to run SparkLens on our cluster.
I switched to the spark_1.6 branch, and also compiled against the specific
image of Spark we are using (cdh5.7.6).
Now I need to figure out what the output means... :P
Shmuel
On Fri, Mar 23, 2018 at 7:24 PM, Fawze Abujaber w
Quick question:
how to add the --jars /path/to/sparklens_2.11-0.1.0.jar to the
spark-default conf, should it be using:
spark.driver.extraClassPath /path/to/sparklens_2.11-0.1.0.jar or i should
use spark.jars option? anyone who could give an example how it should be,
and if i the path for the jar
Hi Shmuel,
Did you compile the code against the right branch for Spark 1.6.
I tested it and it looks working and now i'm testing the branch for a wide
tests, Please use the branch for Spark 1.6
On Fri, Mar 23, 2018 at 12:43 AM, Shmuel Blitz
wrote:
> Hi Rohit,
>
> Thanks for sharing this great
Hi Rohit,
Thanks for sharing this great tool.
I tried running a spark job with the tool, but it failed with an
*IncompatibleClassChangeError
*Exception.
I have opened an issue on Github.(
https://github.com/qubole/sparklens/issues/1)
Shmuel
On Thu, Mar 22, 2018 at 5:05 PM, Shmuel Blitz
wrote:
Thanks.
We will give this a try and report back.
Shmuel
On Thu, Mar 22, 2018 at 4:22 PM, Rohit Karlupia wrote:
> Thanks everyone!
> Please share how it works and how it doesn't. Both help.
>
> Fawaze, just made few changes to make this work with spark 1.6. Can you
> please try building from br
Thanks everyone!
Please share how it works and how it doesn't. Both help.
Fawaze, just made few changes to make this work with spark 1.6. Can you
please try building from branch *spark_1.6*
thanks,
rohitk
On Thu, Mar 22, 2018 at 10:18 AM, Fawze Abujaber wrote:
> It's super amazing i see
ning applications for lower
>>> runtime or cost.
>>>
>>> Please clone from here: https://github.com/qubole/sparklens
>>> Old blogpost: https://www.qubole.com/blog/introducing-quboles-
>>> spark-tuning-tool/
>>>
>>> thanks,
>>> rohitk
>>>
>>> PS: Thanks for the patience. It took couple of months to get back on
>>> this.
>>>
>>>
>>>
>>>
>>>
>> --
> Twitter: https://twitter.com/holdenkarau
>
runtime or cost.
>>
>> Please clone from here: https://github.com/qubole/sparklens
>> Old blogpost:
>> https://www.qubole.com/blog/introducing-quboles-spark-tuning-tool/
>>
>> thanks,
>> rohitk
>>
>> PS: Thanks for the patience. It took couple of months to get back on
>> this.
>>
>>
>>
>>
>>
> --
Twitter: https://twitter.com/holdenkarau
useful guide on the path towards tuning applications for lower
> runtime or cost.
>
> Please clone from here: https://github.com/qubole/sparklens
> Old blogpost: https://www.qubole.com/blog/introducing-quboles-
> spark-tuning-tool/
>
> thanks,
> rohitk
>
> PS: Thanks
/sparklens
Old blogpost: https://www.qubole.com/blog/introducing-quboles-spark-
tuning-tool/
thanks,
rohitk
PS: Thanks for the patience. It took couple of months to get back on this.
upia
> wrote:
> >>
> >> Hi,
> >>
> >> I have been working on making the performance tuning of spark
> applications
> >> bit easier. We have just released the beta version of the tool on
> Qubole.
> >>
> >> https://www.qubole.co
t; I have been working on making the performance tuning of spark applications
>> bit easier. We have just released the beta version of the tool on Qubole.
>>
>> https://www.qubole.com/blog/introducing-quboles-spark-tuning-tool/
>>
>> This is not OSS yet but we would
rsion of the tool on Qubole.
>
> https://www.qubole.com/blog/introducing-quboles-spark-tuning-tool/
>
> This is not OSS yet but we would like to contribute it to OSS. Fishing
> for some interest in the community if people find this work interesting and
> would like to try to it out.
>
> thanks,
> Rohit Karlupia
>
>
>
the beta version of the
>> tool on Qubole.
>>
>> https://www.qubole.com/blog/introducing-quboles-spark-tuning-tool/
>>
>> This is not OSS yet but we would like to contribute it to OSS. Fishing
>> for some interest in the community if people find this work interesting and
>> would like to try to it out.
>>
>> thanks,
>> Rohit Karlupia
>>
>>
>>
>
>
> --
>
>
We have just released the beta version of the tool on Qubole.
>
> https://www.qubole.com/blog/introducing-quboles-spark-tuning-tool/
>
> This is not OSS yet but we would like to contribute it to OSS. Fishing
> for some interest in the community if people find this work interesting an
4:01 pm, "Rohit Karlupia" wrote:
>>>
>>>> Hi,
>>>>
>>>> I have been working on making the performance tuning of spark
>>>> applications bit easier. We have just released the beta version of the
>>>> tool on Qubole.
>>>>
quot; wrote:
>>
>>> Hi,
>>>
>>> I have been working on making the performance tuning of spark
>>> applications bit easier. We have just released the beta version of the
>>> tool on Qubole.
>>>
>>> https://www.qubole.com/blog/intro
> I have been working on making the performance tuning of spark
>> applications bit easier. We have just released the beta version of the
>> tool on Qubole.
>>
>> https://www.qubole.com/blog/introducing-quboles-spark-tuning-tool/
>>
>> This is not OSS yet but we
om/blog/introducing-quboles-spark-tuning-tool/
>
> This is not OSS yet but we would like to contribute it to OSS. Fishing
> for some interest in the community if people find this work interesting and
> would like to try to it out.
>
> thanks,
> Rohit Karlupia
>
>
>
Hi,
I have been working on making the performance tuning of spark applications
bit easier. We have just released the beta version of the tool on Qubole.
https://www.qubole.com/blog/introducing-quboles-spark-tuning-tool/
This is not OSS yet but we would like to contribute it to OSS. Fishing
29 matches
Mail list logo
