I'm trying to get Spark Streaming to scale up/down its number of executors within Mesos based on workload. It's not scaling down. I'm using Spark 1.5.1 reading from Kafka using the direct (receiver-less) approach.
Based on this ticket https://issues.apache.org/jira/browse/SPARK-6287 with the right configuration, I have a simple example working with the spark-shell connected to a Mesos cluster. By working I mean the number of executors scales up/down based on workload. However, the spark-shell is not a streaming example. What is that status of dynamic resource allocation with Spark Streaming on Mesos? Is it supported at all? Or supported but with some caveats to ensure no data loss? thanks, Duc
