[
https://issues.apache.org/jira/browse/SPARK-32701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen updated SPARK-32701:
---------------------------------
Issue Type: Improvement (was: Bug)
Priority: Minor (was: Major)
> mapreduce.fileoutputcommitter.algorithm.version default depends on runtime
> environment
> --------------------------------------------------------------------------------------
>
> Key: SPARK-32701
> URL: https://issues.apache.org/jira/browse/SPARK-32701
> Project: Spark
> Issue Type: Improvement
> Components: docs, Documentation
> Affects Versions: 2.4.0, 3.0.0
> Reporter: Waleed Fateem
> Priority: Minor
>
> When someone reads the documentation in its current state, the assumption is
> that the default value of
> spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version is 1 and that's
> not entirely accurate.
> Spark doesn't explicitly set this configuration and instead is inherited from
> Hadoop's FileOutputCommitter class. The default value is 1 until Hadoop 3.0
> where this changed to 2.
> I'm proposing that we clarify that this value's default will depend on the
> Hadoop version in a user's runtime environment, where:
> 1 for < Hadoop 3.0
> 2 for >= Hadoop 3.0
> There are also plans to revert this default again to v1 so might also be
> useful to reference this JIRA:
> https://issues.apache.org/jira/browse/MAPREDUCE-7282
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
