Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21526#discussion_r201717626
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala
---
@@ -1053,7 +1053,10 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
// users that they may loss data if they are using a direct output
committer.
val speculationEnabled = self.conf.getBoolean("spark.speculation",
false)
val outputCommitterClass =
hadoopConf.get("mapred.output.committer.class", "")
- if (speculationEnabled && outputCommitterClass.contains("Direct")) {
+ val outputCommitCoordinationEnabled = self.conf.getBoolean(
+ "spark.hadoop.outputCommitCoordination.enabled", true)
--- End diff --
since we are touching it, can we define this config in
`org.apache.spark.internal.config`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
