Robert Kruszewski created SPARK-21400:
-----------------------------------------
Summary: Spark shouldn't ignore user defined output committer in
append mode
Key: SPARK-21400
URL: https://issues.apache.org/jira/browse/SPARK-21400
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.2.0
Reporter: Robert Kruszewski
In https://issues.apache.org/jira/browse/SPARK-8578 we decided to override user
defined output committers in append mode. The reasoning was that there's some
output committers that can lead to correctness issues. Since then we have
removed DirectParquetOutputCommitter (the biggest known offender) from codebase
and rely on default implementations.
I believe that we shouldn't be restricting this anymore and users should
understand that if they're overwriting this configuration they have tested
their committer for correctness. This unblocks using more sophisticated and
performant output committers without need to overwrite file format
implementations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
