[
https://issues.apache.org/jira/browse/SPARK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821433#comment-17821433
]
Mich Talebzadeh commented on SPARK-24815:
-----------------------------------------
some thoughts on this if I may
This enhancement request provides a solid foundation for improving dynamic
allocation in Structured Streaming. Adding more specific details, outlining
potential benefits, and addressing potential challenges can further strengthen
the proposal and increase its chances of being implemented.
So these are my thoughts:
- Pluggable Dynamic Allocation: This suggestion shows good design principles,
allowing for flexibility and future improvements. We should elaborate benefits
of a pluggable approach, like customization and integration with external
resource management tools.
- Separate Algorithm for Structured Streaming: This is crucial for adapting
allocation strategies to the unique nature of streaming workloads Also
outlining how a separate algorithm might differ from the batch counterpart
could be useful
- Warning for Enabled Core Dynamic Allocation: This is a valuable warning to
prevent accidental misuse and raise awareness among users. Also consider
suggesting the warning level (e.g. info, warning, error) and potential content
to provide clarity.
- Briefly mention potential challenges or trade-offs associated with
implementing these proposals. Suggesting relevant discussions, resources, or
alternative approaches could strengthen the request for enhancement
> Structured Streaming should support dynamic allocation
> ------------------------------------------------------
>
> Key: SPARK-24815
> URL: https://issues.apache.org/jira/browse/SPARK-24815
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler, Spark Core, Structured Streaming
> Affects Versions: 2.3.1
> Reporter: Karthik Palaniappan
> Priority: Minor
> Labels: pull-request-available
>
> For batch jobs, dynamic allocation is very useful for adding and removing
> containers to match the actual workload. On multi-tenant clusters, it ensures
> that a Spark job is taking no more resources than necessary. In cloud
> environments, it enables autoscaling.
> However, if you set spark.dynamicAllocation.enabled=true and run a structured
> streaming job, the batch dynamic allocation algorithm kicks in. It requests
> more executors if the task backlog is a certain size, and removes executors
> if they idle for a certain period of time.
> Quick thoughts:
> 1) Dynamic allocation should be pluggable, rather than hardcoded to a
> particular implementation in SparkContext.scala (this should be a separate
> JIRA).
> 2) We should make a structured streaming algorithm that's separate from the
> batch algorithm. Eventually, continuous processing might need its own
> algorithm.
> 3) Spark should print a warning if you run a structured streaming job when
> Core's dynamic allocation is enabled
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
