[ https://issues.apache.org/jira/browse/SPARK-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheng Lian resolved SPARK-8990. ------------------------------- Resolution: Fixed Fix Version/s: 1.5.0 1.4.1 Issue resolved by pull request 7351 [https://github.com/apache/spark/pull/7351] > DataFrameReader.parquet() ignores user specified data source options > -------------------------------------------------------------------- > > Key: SPARK-8990 > URL: https://issues.apache.org/jira/browse/SPARK-8990 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.4.0 > Reporter: Cheng Lian > Assignee: Cheng Lian > Fix For: 1.4.1, 1.5.0 > > > A bad consequence of this is that {{sqlContext.read.parquet(path)}} always do > schema merging. For example: > {code} > import sqlContext._ > import sqlContext.implicits._ > val path = "s3n://my-bucket/parquet/tiny" > range(0, 10).coalesce(1).write.mode("overwrite").parquet(path) > // Explicitly disables schema merging > read.option("mergeSchema", "false").format("parquet").load(path) > {code} > However, we still see all files are opened for schema discovery: > {noformat} > 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening > 's3n://databricks-lian/parquet/tiny/_metadata' for reading > 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key > 'parquet/tiny/_metadata' for reading at position '314' > 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening > 's3n://databricks-lian/parquet/tiny/part-r-00000-da490c43-15e2-46b5-95ff-4863e6ab1cc4.gz.parquet' > for reading > 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening > 's3n://databricks-lian/parquet/tiny/_common_metadata' for reading > 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key > 'parquet/tiny/part-r-00000-da490c43-15e2-46b5-95ff-4863e6ab1cc4.gz.parquet' > for reading at position '345' > 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key > 'parquet/tiny/_common_metadata' for reading at position '191' > 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key > 'parquet/tiny/_metadata' for reading at position '4' > 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key > 'parquet/tiny/part-r-00000-da490c43-15e2-46b5-95ff-4863e6ab1cc4.gz.parquet' > for reading at position '97' > 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key > 'parquet/tiny/_common_metadata' for reading at position '4' > {noformat} > To workaround this issue, use the following instead: > {noformat} > sqlContext.read.option("mergeSchema", "false").format("parquet").load(path) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org