[ https://issues.apache.org/jira/browse/SPARK-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheng Lian resolved SPARK-11500. -------------------------------- Resolution: Fixed Fix Version/s: 1.7.0 Issue resolved by pull request 9517 [https://github.com/apache/spark/pull/9517] > Not deterministic order of columns when using merging schemas. > -------------------------------------------------------------- > > Key: SPARK-11500 > URL: https://issues.apache.org/jira/browse/SPARK-11500 > Project: Spark > Issue Type: Bug > Components: SQL > Reporter: Hyukjin Kwon > Assignee: Hyukjin Kwon > Fix For: 1.7.0 > > > When executing > {{sqlContext.read.option("mergeSchema", "true").parquet(pathOne, > pathTwo).printSchema()}} > The order of columns is not deterministic, showing up in a different order > sometimes. > This is because of {{FileStatusCache}} in {{HadoopFsRelation}} (which > {{ParquetRelation}} extends as you know). When > {{FileStatusCache.listLeafFiles()}} is called, this returns > {{Set[FileStatus]}} which messes up the order of {{Array[FileStatus]}}. > So, after retrieving the list of leaf files including {{_metadata}} and > {{_common_metadata}}, this starts to merge (separately and if necessary) the > {{Set}} s of {{_metadata}}, {{_common_metadata}} and part-files in > {{ParquetRelation.mergeSchemasInParallel()}}, which ends up in the different > column order having the leading columns (of the first file) which the other > files do not have. > I think this can be resolved by using {{LinkedHashSet}}. > in a simple view, > If A file has 1,2,3 fields, and B file column 3,4,5, we can not ensure which > column shows first since It is not deterministic. > 1. Read file list (A and B) > 2. Not deterministic order of (A and B or B and A) as I said. > 3. It merges by {{reduceOption}} with retrieved schemas of (A and B or B and > A), (which maybe also should be {{reduceOptionRight}} or > {{reduceOptionLeft}}). > 4. The output columns would be 1,2,3,4,5 when A and B, or 3.4.5.1.2 when B > and A. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org