[jira] [Commented] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

2019-02-08 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763808#comment-16763808 ] Dongjoon Hyun commented on SPARK-24657: --- Hi, [~wangzjie1], [~maropu], [~taoluo]. I close this

[jira] [Commented] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

2019-02-07 Thread Tao Luo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763243#comment-16763243 ] Tao Luo commented on SPARK-24657: - If SortMergeJoinScanner doesn't consume UnsafeExternalRowSorter

[jira] [Commented] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

2019-02-06 Thread Tao Luo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762280#comment-16762280 ] Tao Luo commented on SPARK-24657: - Just reproduced it in standalone mode using 

[jira] [Commented] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

2019-02-06 Thread Tao Luo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761607#comment-16761607 ] Tao Luo commented on SPARK-24657: - Sure: {code:java} from pyspark.sql.functions import rand, col

[jira] [Commented] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

2019-02-04 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760444#comment-16760444 ] Takeshi Yamamuro commented on SPARK-24657: -- Can you put a simple example query for the issue?

[jira] [Commented] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

2019-02-04 Thread Tao Luo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760328#comment-16760328 ] Tao Luo commented on SPARK-24657: - Nice find. This looks like