[jira] [Comment Edited] (SPARK-17380) Spark streaming with a multi shard Kinesis freezes after several days (memory/resource leak?)

[ https://issues.apache.org/jira/browse/SPARK-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824564#comment-15824564 ] Xeto edited comment on SPARK-17380 at 1/16/17 8:41 PM: --- Hi We switc

[jira] [Resolved] (SPARK-17380) Spark streaming with a multi shard Kinesis freezes after several days (memory/resource leak?)

[ https://issues.apache.org/jira/browse/SPARK-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xeto resolved SPARK-17380. -- Resolution: Fixed Fix Version/s: 2.0.2 > Spark streaming with a multi shard Kinesis freezes after severa

[jira] [Commented] (SPARK-17380) Spark streaming with a multi shard Kinesis freezes after several days (memory/resource leak?)

[ https://issues.apache.org/jira/browse/SPARK-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824564#comment-15824564 ] Xeto commented on SPARK-17380: -- Hi We switched to using StorageLevel.MEMORY_AND_DISK_SER on

[jira] [Commented] (SPARK-17379) Upgrade netty-all to 4.0.41.Final (4.1.5-Final not compatible)

[ https://issues.apache.org/jira/browse/SPARK-17379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15470692#comment-15470692 ] Xeto commented on SPARK-17379: -- Hi I'm getting a huge "io.netty.buffer.PoolChunk" leak in Sp

[jira] [Commented] (SPARK-17380) Spark streaming with a multi shard Kinesis freezes after several days (memory/resource leak?)

[ https://issues.apache.org/jira/browse/SPARK-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15470322#comment-15470322 ] Xeto commented on SPARK-17380: -- Decided to try more executors with less memory to reduce the

[jira] [Comment Edited] (SPARK-17380) Spark streaming with a multi shard Kinesis freezes after several days (memory/resource leak?)

[ https://issues.apache.org/jira/browse/SPARK-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15463014#comment-15463014 ] Xeto edited comment on SPARK-17380 at 9/5/16 8:10 AM: -- I simplified

[jira] [Updated] (SPARK-17380) Spark streaming with a multi shard Kinesis freezes after several days (memory/resource leak?)

[ https://issues.apache.org/jira/browse/SPARK-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xeto updated SPARK-17380: - Attachment: exec_Leak_Hunter.zip I simplified the POC on Spark 2.0.0 by removing persist with replicas and group

[jira] [Comment Edited] (SPARK-17380) Spark streaming with a multi shard Kinesis freezes after several days (memory/resource leak?)

[ https://issues.apache.org/jira/browse/SPARK-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15458673#comment-15458673 ] Xeto edited comment on SPARK-17380 at 9/2/16 2:24 PM: -- Attaching Gan

[jira] [Updated] (SPARK-17380) Spark streaming with a multi shard Kinesis freezes after several days (memory/resource leak?)

[ https://issues.apache.org/jira/browse/SPARK-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xeto updated SPARK-17380: - Attachment: memory-after-freeze.png Used memory after Spark has frozen - is finally reduced but doesn't help Spar

[jira] [Commented] (SPARK-17380) Spark streaming with a multi shard Kinesis freezes after several days (memory/resource leak?)

[ https://issues.apache.org/jira/browse/SPARK-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15458669#comment-15458669 ] Xeto commented on SPARK-17380: -- Could you advise how to obtain such an evidence? Not storing

[jira] [Updated] (SPARK-17380) Spark streaming with a multi shard Kinesis freezes after several days (memory/resource leak?)

[ https://issues.apache.org/jira/browse/SPARK-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xeto updated SPARK-17380: - Summary: Spark streaming with a multi shard Kinesis freezes after several days (memory/resource leak?) (was: Spa

[jira] [Updated] (SPARK-17380) Spark streaming with a multi shard Kinesis freezes after several days

[ https://issues.apache.org/jira/browse/SPARK-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xeto updated SPARK-17380: - Priority: Critical (was: Major) Description: Running Spark Streaming 2.0.0 on AWS EMR 5.0.0 consuming fro

[jira] [Updated] (SPARK-17380) Spark streaming with a multi shard Kinesis freezes after several days

[ https://issues.apache.org/jira/browse/SPARK-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xeto updated SPARK-17380: - Attachment: memory.png Attaching Ganglia cluster memory growth graph > Spark streaming with a multi shard Kinesi

[jira] [Created] (SPARK-17380) Spark streaming with a multi shard Kinesis freezes after several days

Xeto created SPARK-17380: Summary: Spark streaming with a multi shard Kinesis freezes after several days Key: SPARK-17380 URL: https://issues.apache.org/jira/browse/SPARK-17380 Project: Spark Issue