[jira] [Commented] (SPARK-21492) Memory leak in SortMergeJoin

[ https://issues.apache.org/jira/browse/SPARK-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829545#comment-16829545 ] Tao Luo commented on SPARK-21492: - The problem is that the task won't complete because of memory being

[jira] [Commented] (SPARK-21492) Memory leak in SortMergeJoin

[ https://issues.apache.org/jira/browse/SPARK-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766598#comment-16766598 ] Tao Luo commented on SPARK-21492: - cc [~tejasp], [~kiszk] for input on code generation to address the

[jira] [Comment Edited] (SPARK-21492) Memory leak in SortMergeJoin

[ https://issues.apache.org/jira/browse/SPARK-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765378#comment-16765378 ] Tao Luo edited comment on SPARK-21492 at 2/12/19 9:35 AM: -- I'll take a stab at

[jira] [Commented] (SPARK-21492) Memory leak in SortMergeJoin

[ https://issues.apache.org/jira/browse/SPARK-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765378#comment-16765378 ] Tao Luo commented on SPARK-21492: - I'll take a stab at this jira, should have something to review today

[jira] [Commented] (SPARK-21492) Memory leak in SortMergeJoin

[ https://issues.apache.org/jira/browse/SPARK-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763825#comment-16763825 ] Tao Luo commented on SPARK-21492: - Can someone add 'affects version' 2.4.0 as well?  > Memory leak in

[jira] [Commented] (SPARK-21492) Memory leak in SortMergeJoin

[ https://issues.apache.org/jira/browse/SPARK-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763820#comment-16763820 ] Tao Luo commented on SPARK-21492: - If SortMergeJoinScanner doesn't consume the iterator from

[jira] [Commented] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763243#comment-16763243 ] Tao Luo commented on SPARK-24657: - If SortMergeJoinScanner doesn't consume UnsafeExternalRowSorter

[jira] [Comment Edited] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762280#comment-16762280 ] Tao Luo edited comment on SPARK-24657 at 2/7/19 1:53 AM: - Just reproduced it in

[jira] [Comment Edited] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761607#comment-16761607 ] Tao Luo edited comment on SPARK-24657 at 2/7/19 2:02 AM: - Sure: {code:java} from

[jira] [Comment Edited] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761607#comment-16761607 ] Tao Luo edited comment on SPARK-24657 at 2/7/19 2:03 AM: - Sure: {code:java} from

[jira] [Commented] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762280#comment-16762280 ] Tao Luo commented on SPARK-24657: - Just reproduced it in standalone mode using 

[jira] [Comment Edited] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761607#comment-16761607 ] Tao Luo edited comment on SPARK-24657 at 2/6/19 9:31 AM: - Sure: {code:java} from

[jira] [Commented] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761607#comment-16761607 ] Tao Luo commented on SPARK-24657: - Sure: {code:java} from pyspark.sql.functions import rand, col

[jira] [Comment Edited] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761607#comment-16761607 ] Tao Luo edited comment on SPARK-24657 at 2/6/19 9:22 AM: - Sure: {code:java} from

[jira] [Comment Edited] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761607#comment-16761607 ] Tao Luo edited comment on SPARK-24657 at 2/6/19 9:21 AM: - Sure: {code:java} from

[jira] [Comment Edited] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760328#comment-16760328 ] Tao Luo edited comment on SPARK-24657 at 2/5/19 12:25 AM: -- Nice find. This

[jira] [Comment Edited] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760328#comment-16760328 ] Tao Luo edited comment on SPARK-24657 at 2/5/19 12:05 AM: -- Nice find. This

[jira] [Commented] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760328#comment-16760328 ] Tao Luo commented on SPARK-24657: - Nice find. This looks like