[jira] [Commented] (SPARK-7661) Support for dynamic allocation of executors in Kinesis Spark Streaming
[ https://issues.apache.org/jira/browse/SPARK-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547642#comment-14547642 ] Murtaza Kanchwala commented on SPARK-7661: -- Yes, it works I took 4 + 4 = 8 cores, where 4 is my total no. of cores and 4 is my total no .of shards. But there is still an another thing, Now If the Spark Consumer starts up it takes 1 executor and 4 Network Receiver, when I Scale up my Kinesis Stream by 4 more shards,i.e. 8 Shards, then it also works but the Receiver count is still 4. So is there any way to scale up the receivers or not? Support for dynamic allocation of executors in Kinesis Spark Streaming -- Key: SPARK-7661 URL: https://issues.apache.org/jira/browse/SPARK-7661 Project: Spark Issue Type: New Feature Components: Streaming Affects Versions: 1.3.1 Environment: AWS-EMR Reporter: Murtaza Kanchwala Currently the no. of cores is (N + 1), where N is no. of shards in a Kinesis Stream. My Requirement is that if I use this Resharding util for Amazon Kinesis : Amazon Kinesis Resharding : https://github.com/awslabs/amazon-kinesis-scaling-utils Then there should be some way to allocate executors on the basis of no. of shards directly (for Spark Streaming only). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7661) Support for dynamic allocation of executors in Kinesis Spark Streaming
[ https://issues.apache.org/jira/browse/SPARK-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14546617#comment-14546617 ] Tathagata Das commented on SPARK-7661: -- N+1 is used in the example, but isnt really the suggested recommended way. Here is how it works. You have to give X + Y cores, where X = number of Kinesis streams/receivers and Y = number of cores for processing the data. The X receivers will in collaboration with each other receive data from N shards. If you expect your N to vary from 10 to 20, then having X = 15 isnt a bad idea. At N = 20, the 15 receivers wil distribute the work among themselves. And Y should be such that your systems can process the data as fast as it is received. Support for dynamic allocation of executors in Kinesis Spark Streaming -- Key: SPARK-7661 URL: https://issues.apache.org/jira/browse/SPARK-7661 Project: Spark Issue Type: New Feature Components: Streaming Affects Versions: 1.3.1 Environment: AWS-EMR Reporter: Murtaza Kanchwala Currently the no. of cores is (N + 1), where N is no. of shards in a Kinesis Stream. My Requirement is that if I use this Resharding util for Amazon Kinesis : Amazon Kinesis Resharding : https://github.com/awslabs/amazon-kinesis-scaling-utils Then there should be some way to allocate executors on the basis of no. of shards directly (for Spark Streaming only). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7661) Support for dynamic allocation of executors in Kinesis Spark Streaming
[ https://issues.apache.org/jira/browse/SPARK-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14546623#comment-14546623 ] Murtaza Kanchwala commented on SPARK-7661: -- Ok, Let me try your solution as well with this scaling util. Support for dynamic allocation of executors in Kinesis Spark Streaming -- Key: SPARK-7661 URL: https://issues.apache.org/jira/browse/SPARK-7661 Project: Spark Issue Type: New Feature Components: Streaming Affects Versions: 1.3.1 Environment: AWS-EMR Reporter: Murtaza Kanchwala Currently the no. of cores is (N + 1), where N is no. of shards in a Kinesis Stream. My Requirement is that if I use this Resharding util for Amazon Kinesis : Amazon Kinesis Resharding : https://github.com/awslabs/amazon-kinesis-scaling-utils Then there should be some way to allocate executors on the basis of no. of shards directly (for Spark Streaming only). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7661) Support for dynamic allocation of executors in Kinesis Spark Streaming
[ https://issues.apache.org/jira/browse/SPARK-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14546598#comment-14546598 ] Murtaza Kanchwala commented on SPARK-7661: -- Ok I'll correct my terms, My case is exactly like this https://mail-archives.apache.org/mod_mbox/spark-user/201412.mbox/%3c30d8e3e3-95db-492b-8b49-73a99d587...@gmail.com%3E Just the difference is that my no. of spark shards are updating by this utility provided by AWS, and when the no. of shards increase my Spark streaming consumer got's hunged up and and it goes in the waiting state Support for dynamic allocation of executors in Kinesis Spark Streaming -- Key: SPARK-7661 URL: https://issues.apache.org/jira/browse/SPARK-7661 Project: Spark Issue Type: New Feature Components: Streaming Affects Versions: 1.3.1 Environment: AWS-EMR Reporter: Murtaza Kanchwala Currently the no. of cores is (N + 1), where N is no. of shards in a Kinesis Stream. My Requirement is that if I use this Resharding util for Amazon Kinesis : Amazon Kinesis Resharding : https://github.com/awslabs/amazon-kinesis-scaling-utils Then there should be some way to allocate executors on the basis of no. of shards directly (for Spark Streaming only). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7661) Support for dynamic allocation of executors in Kinesis Spark Streaming
[ https://issues.apache.org/jira/browse/SPARK-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14546216#comment-14546216 ] Tathagata Das commented on SPARK-7661: -- What do you mean by the currently the logic is N+1 executor? Is that documented somewhere that you have to have exactly N+1 ? Support for dynamic allocation of executors in Kinesis Spark Streaming -- Key: SPARK-7661 URL: https://issues.apache.org/jira/browse/SPARK-7661 Project: Spark Issue Type: New Feature Components: Streaming Affects Versions: 1.3.1 Environment: AWS-EMR Reporter: Murtaza Kanchwala Currently the logic for the no. of executors is (N + 1), where N is no. of shards in a Kinesis Stream. My Requirement is that if I use this Resharding util for Amazon Kinesis : Amazon Kinesis Resharding : https://github.com/awslabs/amazon-kinesis-scaling-utils Then there should be some way to allocate executors on the basis of no. of shards directly (for Spark Streaming only). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
