[ https://issues.apache.org/jira/browse/SPARK-8121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheng Lian updated SPARK-8121: ------------------------------ Description: When {{spark.sql.sources.outputCommitterClass}} is configured, {{spark.sql.parquet.output.committer.class}} will be overriden. For example, if {{spark.sql.parquet.output.committer.class}} is set to {{FileOutputCommitter}}, while {{spark.sql.sources.outputCommitterClass}} is set to {{DirectParquetOutputCommitter}}, neither {{_metadata}} nor {{_common_metadata}} will be written because {{FileOutputCommitter}} overrides {{DirectParquetOutputCommitter}}. was: When {{spark.sql.sources.outputCommitterClass}} is configured, {{spark.sql.parquet.output.committer.class}} will be overriden. For example, if {{spark.sql.parquet.output.committer.class}} is set to {{FileOutputCommitter}}, while {{spark.sql.sources.outputCommitterClass}} is set to {{DirectParquetOutputCommitter}}, neither {{_metadata}} nor {{_common_metadata}} will be written because {{FileOutputCommitter}} overrides {{DirectParquetOutputCommitter}}. The reason is that, in > "spark.sql.parquet.output.committer.class" is overriden by > "spark.sql.sources.outputCommitterClass" > --------------------------------------------------------------------------------------------------- > > Key: SPARK-8121 > URL: https://issues.apache.org/jira/browse/SPARK-8121 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.4.0 > Reporter: Cheng Lian > Assignee: Cheng Lian > > When {{spark.sql.sources.outputCommitterClass}} is configured, > {{spark.sql.parquet.output.committer.class}} will be overriden. > For example, if {{spark.sql.parquet.output.committer.class}} is set to > {{FileOutputCommitter}}, while {{spark.sql.sources.outputCommitterClass}} is > set to {{DirectParquetOutputCommitter}}, neither {{_metadata}} nor > {{_common_metadata}} will be written because {{FileOutputCommitter}} > overrides {{DirectParquetOutputCommitter}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org