Dynamic allocation doesn't work yet with Spark Streaming in any cluster scenario. There was a previous thread on this topic which discusses the issues that need to be resolved.
Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Typesafe <http://typesafe.com> @deanwampler <http://twitter.com/deanwampler> http://polyglotprogramming.com On Wed, Nov 11, 2015 at 8:09 AM, PhuDuc Nguyen <duc.was.h...@gmail.com> wrote: > I'm trying to get Spark Streaming to scale up/down its number of executors > within Mesos based on workload. It's not scaling down. I'm using Spark > 1.5.1 reading from Kafka using the direct (receiver-less) approach. > > Based on this ticket https://issues.apache.org/jira/browse/SPARK-6287 > with the right configuration, I have a simple example working with the > spark-shell connected to a Mesos cluster. By working I mean the number of > executors scales up/down based on workload. However, the spark-shell is not > a streaming example. > > What is that status of dynamic resource allocation with Spark Streaming on > Mesos? Is it supported at all? Or supported but with some caveats to ensure > no data loss? > > thanks, > Duc >
