[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931619#comment-15931619 ] Ran Haim commented on SPARK-17436: -- Hi, I think we need to reopen this. It seems that on

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836847#comment-15836847 ] Jason Moore commented on SPARK-17436: - Ahh, I think you are correct. The issue on the write seems to

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833367#comment-15833367 ] Ran Haim commented on SPARK-17436: -- Hi, I did not actually use 2.1 yet - so I cannot be 100% sure. I can

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830889#comment-15830889 ] Jason Moore commented on SPARK-17436: - Hi [~ran.h...@optimalplus.com], [~srowen], How sure are we

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15680994#comment-15680994 ] Ran Haim commented on SPARK-17436: -- Sure. Basically I propose to stop using UnsafeKVExternalSorter, and

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15679303#comment-15679303 ] Sean Owen commented on SPARK-17436: --- Opening a pull request is available to everyone. Maybe you are

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15679204#comment-15679204 ] Ran Haim commented on SPARK-17436: -- Hi, When you want to write your data to orc files or perquet files,

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15679106#comment-15679106 ] Sean Owen commented on SPARK-17436: --- [~ran.h...@optimalplus.com] I'm not sure how to proceed on this.

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674253#comment-15674253 ] Sean Owen commented on SPARK-17436: --- Not sure, I have no failures on OS X, on Ubuntu, and all of the

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674032#comment-15674032 ] Ran Haim commented on SPARK-17436: -- I have basiaclly cloned the repository from

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673230#comment-15673230 ] Ran Haim commented on SPARK-17436: -- I am running thee build on linux, so this is not it. It is something

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664352#comment-15664352 ] Sean Owen commented on SPARK-17436: --- See recent JIRAs about JavaAPISuite on Windows. I assume you might

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664233#comment-15664233 ] Ran Haim commented on SPARK-17436: -- So... Can you help? Or at least point me to someone who can? BTW On

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663358#comment-15663358 ] Ran Haim commented on SPARK-17436: -- Hey, I'll do a pull and try again How to I gain access to create

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663348#comment-15663348 ] Sean Owen commented on SPARK-17436: --- I don't have any problems building or making PRs now, and Jenkins

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663345#comment-15663345 ] Ran Haim commented on SPARK-17436: -- Look, I think this is a serious bug - it makes parquet files and

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663343#comment-15663343 ] Ran Haim commented on SPARK-17436: -- but the whole thing is that the partitioning here actually happens a

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661640#comment-15661640 ] Ran Haim commented on SPARK-17436: -- Hi, I only got a chance to work on it now. I saw that the whole

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611483#comment-15611483 ] Ran Haim commented on SPARK-17436: -- usually you partition the data, and then you order it - this way you

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611466#comment-15611466 ] Sean Owen commented on SPARK-17436: --- Yes, but a partitioning operation does not necessarily preserve

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611457#comment-15611457 ] Ran Haim commented on SPARK-17436: -- Of course it does, every technology that supports partitioning

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611350#comment-15611350 ] Sean Owen commented on SPARK-17436: --- I don't think partitioning is supposed to maintain ordering. >

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611331#comment-15611331 ] Ran Haim commented on SPARK-17436: -- anyone? > dataframe.write sometimes does not keep sorting >

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15471291#comment-15471291 ] Ran Haim commented on SPARK-17436: -- I tries it under 1.6.1, but I did not see anything that fixes that