[ https://issues.apache.org/jira/browse/SPARK-20144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950988#comment-15950988 ]
Sean Owen commented on SPARK-20144: ----------------------------------- If the data were sorted, sorting would be pretty cheap, in general. Correctness has to take precedence in any event, if you're describing this as a blocker for you. I don't believe projection can change ordering, no. I am saying that I would not necessarily expect that to extend to external serialization. I don't see that being tabular or on HDFS matters. I think some serializations would naturally preserve order and others would not. I am still not 100% sure what the expected semantics of Parquet are here, but you have de facto evidence it is not guaranteed. > spark.read.parquet no long maintains ordering of the data > --------------------------------------------------------- > > Key: SPARK-20144 > URL: https://issues.apache.org/jira/browse/SPARK-20144 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.0.2 > Reporter: Li Jin > > Hi, We are trying to upgrade Spark from 1.6.3 to 2.0.2. One issue we found is > when we read parquet files in 2.0.2, the ordering of rows in the resulting > dataframe is not the same as the ordering of rows in the dataframe that the > parquet file was reproduced with. > This is because FileSourceStrategy.scala combines the parquet files into > fewer partitions and also reordered them. This breaks our workflows because > they assume the ordering of the data. > Is this considered a bug? Also FileSourceStrategy and FileSourceScanExec > changed quite a bit from 2.0.2 to 2.1, so not sure if this is an issue with > 2.1. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org