Re: [Spark Structured Streaming]: Dynamic Scaling of Executors

[Aishwarya Panicker]

Hi,


Thanks for your response.


I understand there is no explicit way to configure dynamic scaling for
Spark Structured Streaming as the ticket is still open for that. But is
there a way to manage dynamic scaling with the existing Batch Dynamic
scaling algorithm as this kicks in when Dynamic allocation is enabled with
Structured Streaming. The issue I’m facing with batch dynamic allocation is
that it requests executors based on pending/running tasks. And to have
parallelism we have set spark.sql.shuffle.partitions: "100"  due to which
100 partitions are getting created and thus 100 tasks which is causing more
executors to be requested(not scaling based on load). Is there mechanism to
control this autoscaling behaviour of executors based on data load?


Additionally, Spark Streaming dynamic allocation algorithm autoscales
executors based on the processing time/ batch interval ratio which would be
a preferred method for streaming use case. So is there a provision to use
the streaming configurations instead of the batch mode configurations with
structured streaming?


Any suggestions on the above would be helpful.


Thanks and Regards,

Aishwarya


On Thu, 25 May, 2023, 11:46 PM Mich Talebzadeh, 
wrote:

> Hi,
> Autoscaling is not compatible with Spark Structured Streaming
> 
>  since
> Spark Structured Streaming currently does not support dynamic allocation
> (see SPARK-24815: Structured Streaming should support dynamic allocation
> ).
>
> That ticket is still open
>
> HTH
>
> Mich Talebzadeh,
> Lead Solutions Architect/Engineering Lead
> Palantir Technologies Limited
> London
> United Kingdom
>
>
>view my Linkedin profile
> 
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Thu, 25 May 2023 at 18:44, Aishwarya Panicker <
> aishwaryapanicke...@gmail.com> wrote:
>
>> Hi Team,
>>
>> I have been working on Spark Structured Streaming and trying to autoscale
>> our application through dynamic allocation. But I couldn't find any
>> documentation or configurations that supports dynamic scaling in Spark
>> Structured Streaming, due to which I had been using Spark Batch mode
>> dynamic scaling which is not so efficient with streaming use case.
>>
>> I also tried with Spark streaming dynamic allocation configurations which
>> didn't work with structured streaming.
>>
>> Below are the configurations I tried for dynamic scaling of my Spark
>> Structured Streaming Application:
>>
>> With Batch Spark configurations:
>>
>> spark.dynamicAllocation.enabled: true
>> spark.dynamicAllocation.executorAllocationRatio: 0.5
>> spark.dynamicAllocation.minExecutors: 1
>> spark.dynamicAllocation.maxExecutors: 5
>>
>>
>> With Streaming Spark configurations:
>>
>> spark.dynamicAllocation.enabled: false
>> spark.streaming.dynamicAllocation.enabled: true
>> spark.streaming.dynamicAllocation.scaleUpRatio: 0.7
>> spark.streaming.dynamicAllocation.scaleDownRatio: 0.2
>> spark.streaming.dynamicAllocation.minExecutors: 1
>> spark.streaming.dynamicAllocation.maxExecutors: 5
>>
>> Kindly let me know if there is any configuration for the dynamic
>> allocation of Spark Structured Streaming which I'm missing due to which
>> autoscaling of my application is not working properly.
>>
>> Awaiting your response.
>>
>> Thanks and Regards,
>> Aishwarya
>>
>>
>>
>>
>>

Re: [Spark Structured Streaming]: Dynamic Scaling of Executors

[Mich Talebzadeh]

Hi,
Autoscaling is not compatible with Spark Structured Streaming

since
Spark Structured Streaming currently does not support dynamic allocation
(see SPARK-24815: Structured Streaming should support dynamic allocation
).

That ticket is still open

HTH

Mich Talebzadeh,
Lead Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Thu, 25 May 2023 at 18:44, Aishwarya Panicker <
aishwaryapanicke...@gmail.com> wrote:

> Hi Team,
>
> I have been working on Spark Structured Streaming and trying to autoscale
> our application through dynamic allocation. But I couldn't find any
> documentation or configurations that supports dynamic scaling in Spark
> Structured Streaming, due to which I had been using Spark Batch mode
> dynamic scaling which is not so efficient with streaming use case.
>
> I also tried with Spark streaming dynamic allocation configurations which
> didn't work with structured streaming.
>
> Below are the configurations I tried for dynamic scaling of my Spark
> Structured Streaming Application:
>
> With Batch Spark configurations:
>
> spark.dynamicAllocation.enabled: true
> spark.dynamicAllocation.executorAllocationRatio: 0.5
> spark.dynamicAllocation.minExecutors: 1
> spark.dynamicAllocation.maxExecutors: 5
>
>
> With Streaming Spark configurations:
>
> spark.dynamicAllocation.enabled: false
> spark.streaming.dynamicAllocation.enabled: true
> spark.streaming.dynamicAllocation.scaleUpRatio: 0.7
> spark.streaming.dynamicAllocation.scaleDownRatio: 0.2
> spark.streaming.dynamicAllocation.minExecutors: 1
> spark.streaming.dynamicAllocation.maxExecutors: 5
>
> Kindly let me know if there is any configuration for the dynamic
> allocation of Spark Structured Streaming which I'm missing due to which
> autoscaling of my application is not working properly.
>
> Awaiting your response.
>
> Thanks and Regards,
> Aishwarya
>
>
>
>
>

[Spark Structured Streaming]: Dynamic Scaling of Executors

[Aishwarya Panicker]

Hi Team,

I have been working on Spark Structured Streaming and trying to autoscale
our application through dynamic allocation. But I couldn't find any
documentation or configurations that supports dynamic scaling in Spark
Structured Streaming, due to which I had been using Spark Batch mode
dynamic scaling which is not so efficient with streaming use case.

I also tried with Spark streaming dynamic allocation configurations which
didn't work with structured streaming.

Below are the configurations I tried for dynamic scaling of my Spark
Structured Streaming Application:

With Batch Spark configurations:

spark.dynamicAllocation.enabled: true
spark.dynamicAllocation.executorAllocationRatio: 0.5
spark.dynamicAllocation.minExecutors: 1
spark.dynamicAllocation.maxExecutors: 5


With Streaming Spark configurations:

spark.dynamicAllocation.enabled: false
spark.streaming.dynamicAllocation.enabled: true
spark.streaming.dynamicAllocation.scaleUpRatio: 0.7
spark.streaming.dynamicAllocation.scaleDownRatio: 0.2
spark.streaming.dynamicAllocation.minExecutors: 1
spark.streaming.dynamicAllocation.maxExecutors: 5

Kindly let me know if there is any configuration for the dynamic allocation
of Spark Structured Streaming which I'm missing due to which autoscaling of
my application is not working properly.

Awaiting your response.

Thanks and Regards,
Aishwarya