[ https://issues.apache.org/jira/browse/SPARK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850235#comment-16850235 ]
Jungtaek Lim commented on SPARK-24815: -------------------------------------- I'm also interested on the design doc, as I'd like to see whether the fact is considered as well: there're some points which dynamic allocation could hurt the performance of streaming query. Executors running structured streaming query is stateful, at least for stateful queries, and query leveraging Kafka source. > Structured Streaming should support dynamic allocation > ------------------------------------------------------ > > Key: SPARK-24815 > URL: https://issues.apache.org/jira/browse/SPARK-24815 > Project: Spark > Issue Type: Improvement > Components: Scheduler, Structured Streaming > Affects Versions: 2.3.1 > Reporter: Karthik Palaniappan > Priority: Minor > > Dynamic allocation is very useful for adding and removing containers to match > the actual workload. On multi-tenant clusters, it ensures that a Spark job is > taking no more resources than necessary. In cloud environments, it enables > autoscaling. > However, if you set spark.dynamicAllocation.enabled=true and run a structured > streaming job, Core's dynamic allocation algorithm kicks in. It requests > executors if the task backlog is a certain size, and remove executors if they > idle for a certain period of time. > This does not make sense for streaming jobs, as outlined in > https://issues.apache.org/jira/browse/SPARK-12133, which introduced dynamic > allocation for the old streaming API. > First, Spark should print a warning if you run a structured streaming job > when Core's dynamic allocation is enabled > Second, structured streaming should have support for dynamic allocation. It > would be convenient if it were the same set of properties as Core's dynamic > allocation, but I don't have a strong opinion on that. > If somebody can give me pointers on how to add dynamic allocation support, > I'd be happy to take a stab. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org