[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2017-02-13 Thread Deenbandhu Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865229#comment-15865229 ] Deenbandhu Agarwal commented on SPARK-15716: What happened to this issue ? > Memory usage

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-14 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330526#comment-15330526 ] Yan Chen commented on SPARK-15716: -- The checkpoint path is on HDFS. The application is already shutdown,

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-14 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330497#comment-15330497 ] Shixiong Zhu commented on SPARK-15716: -- I saw there were a lot of

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-14 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15329990#comment-15329990 ] Yan Chen commented on SPARK-15716: -- !http://i.imgur.com/gcm4Y6p.png! I have difficulties uploading

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323019#comment-15323019 ] Sean Owen commented on SPARK-15716: --- I agree you obviously have some issue in your system. The question

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323004#comment-15323004 ] Yan Chen commented on SPARK-15716: -- So you agree that it is a problem. I also agree that it is still in

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322989#comment-15322989 ] Yan Chen commented on SPARK-15716: -- Will do it in a few days when I get a chance. > Memory usage of

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322990#comment-15322990 ] Sean Owen commented on SPARK-15716: --- I wouldn't go that far. Without a pretty clear reproduction (or

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322982#comment-15322982 ] Shixiong Zhu commented on SPARK-15716: -- If possible, could you provide your heap dump based on the

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322959#comment-15322959 ] Yan Chen commented on SPARK-15716: -- I've marked it as invalid since it is a problem, but you think it is

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322950#comment-15322950 ] Yan Chen commented on SPARK-15716: -- Does it mean if we can not identify what the problem is, we should

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322938#comment-15322938 ] Sean Owen commented on SPARK-15716: --- [~yani.chen] the problem is you've described several different

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322942#comment-15322942 ] Sean Owen commented on SPARK-15716: --- [~yani.chen] the problem is you've described several different

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322930#comment-15322930 ] Yan Chen commented on SPARK-15716: -- Why it is "not a problem" even if it crashes the streaming process?

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322920#comment-15322920 ] Yan Chen commented on SPARK-15716: -- [~srowen] Could I know why this issue is closed? > Memory usage of

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322912#comment-15322912 ] Yan Chen commented on SPARK-15716: -- Original problem comes from Hortonworks. We also tried to use

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-06 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317340#comment-15317340 ] Shixiong Zhu commented on SPARK-15716: -- [~yani.chen] Are you using the Apache Spark or other

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-06 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316539#comment-15316539 ] Yan Chen commented on SPARK-15716: -- I'm not sure what is causing that behavior, but that is what we saw.

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316325#comment-15316325 ] Sean Owen commented on SPARK-15716: --- Hm, but this is unrelated to memory. An application that runs out

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-05 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316050#comment-15316050 ] Yan Chen commented on SPARK-15716: -- We think the behavior of using local directory is due to this piece

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-05 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316047#comment-15316047 ] Yan Chen commented on SPARK-15716: -- The memory usage goes up, and finally will make the whole streaming

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-05 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316045#comment-15316045 ] Yan Chen commented on SPARK-15716: -- The same thing happens on 1.6.0, 1.6.1, 2.0.0 preview. However, if I

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-05 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315972#comment-15315972 ] Yan Chen commented on SPARK-15716: -- Actually I tried in another environment with version 1.6.1, the same

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315895#comment-15315895 ] Sean Owen commented on SPARK-15716: --- [~HalfVim] are you experiencing out of memory errors? it's normal

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-05 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315890#comment-15315890 ] Yan Chen commented on SPARK-15716: -- We already tried 1.6.0, it does not happen. Could I know what

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-03 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314908#comment-15314908 ] Shixiong Zhu commented on SPARK-15716: -- [~yani.chen] could you try 1.6.1 and see if this still

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-03 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314190#comment-15314190 ] Yan Chen commented on SPARK-15716: -- I also tried to run scala code adapted from the example code in