[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-20 Thread Jianfei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508257#comment-15508257 ] Jianfei Wang commented on SPARK-17562: -- that sounds reasonable, thank you Josh Rosen! > I think a

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-20 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508076#comment-15508076 ] Josh Rosen commented on SPARK-17562: I'm not sure that "zero objects written" necessarily implies "no

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-17 Thread Jianfei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500059#comment-15500059 ] Jianfei Wang commented on SPARK-17562: -- [~joshrosen] please check this. thank you very much! > I

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Jianfei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496324#comment-15496324 ] Jianfei Wang commented on SPARK-17562: -- 中秋快乐！谢谢。 > I think a little code is unnecessary to exist in

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Jianfei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496323#comment-15496323 ] Jianfei Wang commented on SPARK-17562: -- 中秋快乐！谢谢。 > I think a little code is unnecessary to exist in

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496302#comment-15496302 ] Wenchen Fan commented on SPARK-17562: - I'm not familiar with this part, cc [~joshrosen] > I think a

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Jianfei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496283#comment-15496283 ] Jianfei Wang commented on SPARK-17562: -- this func is to revert writes that haven't been committed

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496251#comment-15496251 ] Sean Owen commented on SPARK-17562: --- If you do that, you don't call revertPartialWritesAndClose() then.

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Jianfei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496183#comment-15496183 ] Jianfei Wang commented on SPARK-17562: -- if 0 object is written, we should just set the success flag

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496167#comment-15496167 ] Sean Owen commented on SPARK-17562: --- Can you explain why? it looks possible that 0 objects are written.

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Jianfei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496147#comment-15496147 ] Jianfei Wang commented on SPARK-17562: -- [~cloud_fan] can you check this? thank you! > I think a