Re: dynamic allocation manager in SS

Hello, FYI, there are SPARK-24815 JIRA for adding support for Dynamic allocation support in Spark Streaming. We plan to work on this over the summer. Let's move design discussion to the JIRA so it will be easier to move it forward. Best

Re: dynamic allocation manager in SS

Sure im not talking about k8s here. The discussion is about the heuristics and their drawbacks. Στις Δευ, 27 Μαΐ 2019, 2:04 μ.μ. ο χρήστης Gabor Somogyi < gabor.g.somo...@gmail.com> έγραψε: > K8s is a different story, please take a look at the doc "Future Work" part. > > On Fri, May 24, 2019 at

Re: dynamic allocation manager in SS

K8s is a different story, please take a look at the doc "Future Work" part. On Fri, May 24, 2019 at 9:40 PM Stavros Kontopoulos < stavros.kontopou...@lightbend.com> wrote: > Btw the heuristics for batch mode ( >

Re: dynamic allocation manager in SS

Btw the heuristics for batch mode ( https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L289) vs streaming (

Re: dynamic allocation manager in SS

I am on k8s where there is no support yet afaik, there is wip wrt the shuffle service. So from your experience there are no issues with using the batch dynamic allocation version like there was before with dstreams as described in the related jira? Στις Παρ, 24 Μαΐ 2019, 8:28 μ.μ. ο χρήστης Gabor

Re: dynamic allocation manager in SS

Yes nothing happens. In this case it could propagate info to the resource manager to scale down the number of executors no? Just a thought. Στις Παρ, 24 Μαΐ 2019, 7:17 μ.μ. ο χρήστης Gabor Somogyi < gabor.g.somo...@gmail.com> έγραψε: > Structured Streaming works differently. If no data arrives

Re: dynamic allocation manager in SS

It scales down with yarn. Not sure how you've tested. On Fri, 24 May 2019, 19:10 Stavros Kontopoulos, < stavros.kontopou...@lightbend.com> wrote: > Yes nothing happens. In this case it could propagate info to the resource > manager to scale down the number of executors no? Just a thought. > >

Re: dynamic allocation manager in SS

Structured Streaming works differently. If no data arrives no tasks are executed (just had a case in this area). BR, G On Fri, 24 May 2019, 18:14 Stavros Kontopoulos, < stavros.kontopou...@lightbend.com> wrote: > Hi, > > Some while ago the streaming dynamic allocation part was added in

dynamic allocation manager in SS

Hi, Some while ago the streaming dynamic allocation part was added in DStreams( https://issues.apache.org/jira/browse/SPARK-12133) to improve the issues with the batch based one. Should this be ported to structured streaming? Thoughts? AFAIK there is no support in SS for it. Best, Stavros