[
https://issues.apache.org/jira/browse/SPARK-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546598#comment-14546598
]
Murtaza Kanchwala commented on SPARK-7661:
------------------------------------------
Ok I'll correct my terms, My case is exactly like this
https://mail-archives.apache.org/mod_mbox/spark-user/201412.mbox/%3c30d8e3e3-95db-492b-8b49-73a99d587...@gmail.com%3E
Just the difference is that my no. of spark shards are updating by this utility
provided by AWS, and when the no. of shards increase my Spark streaming
consumer got's hunged up and and it goes in the waiting state
> Support for dynamic allocation of executors in Kinesis Spark Streaming
> ----------------------------------------------------------------------
>
> Key: SPARK-7661
> URL: https://issues.apache.org/jira/browse/SPARK-7661
> Project: Spark
> Issue Type: New Feature
> Components: Streaming
> Affects Versions: 1.3.1
> Environment: AWS-EMR
> Reporter: Murtaza Kanchwala
>
> Currently the no. of cores is (N + 1), where N is no. of shards in a Kinesis
> Stream.
> My Requirement is that if I use this Resharding util for Amazon Kinesis :
> Amazon Kinesis Resharding :
> https://github.com/awslabs/amazon-kinesis-scaling-utils
> Then there should be some way to allocate executors on the basis of no. of
> shards directly (for Spark Streaming only).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
