[spark] 01/01: Preparing development version 3.0.2-SNAPSHOT

[ruifengz]

This is an automated email from the ASF dual-hosted git repository.

ruifengz pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git

commit ed7be8ca4835fc30b54b92c6862ccd549917dfce
Author: zhengruifeng 
AuthorDate: Fri Aug 28 03:25:21 2020 +

Preparing development version 3.0.2-SNAPSHOT
---
 R/pkg/DESCRIPTION  | 2 +-
 assembly/pom.xml   | 2 +-
 common/kvstore/pom.xml | 2 +-
 common/network-common/pom.xml  | 2 +-
 common/network-shuffle/pom.xml | 2 +-
 common/network-yarn/pom.xml| 2 +-
 common/sketch/pom.xml  | 2 +-
 common/tags/pom.xml| 2 +-
 common/unsafe/pom.xml  | 2 +-
 core/pom.xml   | 2 +-
 docs/_config.yml   | 4 ++--
 examples/pom.xml   | 2 +-
 external/avro/pom.xml  | 2 +-
 external/docker-integration-tests/pom.xml  | 2 +-
 external/kafka-0-10-assembly/pom.xml   | 2 +-
 external/kafka-0-10-sql/pom.xml| 2 +-
 external/kafka-0-10-token-provider/pom.xml | 2 +-
 external/kafka-0-10/pom.xml| 2 +-
 external/kinesis-asl-assembly/pom.xml  | 2 +-
 external/kinesis-asl/pom.xml   | 2 +-
 external/spark-ganglia-lgpl/pom.xml| 2 +-
 graphx/pom.xml | 2 +-
 hadoop-cloud/pom.xml   | 2 +-
 launcher/pom.xml   | 2 +-
 mllib-local/pom.xml| 2 +-
 mllib/pom.xml  | 2 +-
 pom.xml| 2 +-
 python/pyspark/version.py  | 2 +-
 repl/pom.xml   | 2 +-
 resource-managers/kubernetes/core/pom.xml  | 2 +-
 resource-managers/kubernetes/integration-tests/pom.xml | 2 +-
 resource-managers/mesos/pom.xml| 2 +-
 resource-managers/yarn/pom.xml | 2 +-
 sql/catalyst/pom.xml   | 2 +-
 sql/core/pom.xml   | 2 +-
 sql/hive-thriftserver/pom.xml  | 2 +-
 sql/hive/pom.xml   | 2 +-
 streaming/pom.xml  | 2 +-
 tools/pom.xml  | 2 +-
 39 files changed, 40 insertions(+), 40 deletions(-)

diff --git a/R/pkg/DESCRIPTION b/R/pkg/DESCRIPTION
index 86514f2..a17dea9 100644
--- a/R/pkg/DESCRIPTION
+++ b/R/pkg/DESCRIPTION
@@ -1,6 +1,6 @@
 Package: SparkR
 Type: Package
-Version: 3.0.1
+Version: 3.0.2
 Title: R Front End for 'Apache Spark'
 Description: Provides an R Front end for 'Apache Spark' 
.
 Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"),
diff --git a/assembly/pom.xml b/assembly/pom.xml
index c47425a..e265bb9 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -21,7 +21,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.1
+3.0.2-SNAPSHOT
 ../pom.xml
   
 
diff --git a/common/kvstore/pom.xml b/common/kvstore/pom.xml
index 0948c6b..f433764 100644
--- a/common/kvstore/pom.xml
+++ b/common/kvstore/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.1
+3.0.2-SNAPSHOT
 ../../pom.xml
   
 
diff --git a/common/network-common/pom.xml b/common/network-common/pom.xml
index e7b16f1..4f31606 100644
--- a/common/network-common/pom.xml
+++ b/common/network-common/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.1
+3.0.2-SNAPSHOT
 ../../pom.xml
   
 
diff --git a/common/network-shuffle/pom.xml b/common/network-shuffle/pom.xml
index c32a74f..a3ac661 100644
--- a/common/network-shuffle/pom.xml
+++ b/common/network-shuffle/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.1
+3.0.2-SNAPSHOT
 ../../pom.xml
   
 
diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml
index 56ca725..5c5dfba 100644
--- a/common/network-yarn/pom.xml
+++ b/common/network-yarn/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.1
+3.0.2-SNAPSHOT
 ../../pom.xml
   
 
diff --git a/common/sketch/pom.xml b/common/sketch/pom.xml
index 6b60cb3..8041d7a 100644
--- a/common/sketch/pom.xml
+++ b/common/sketch/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.1
+3.0.2-SNAPSHOT
 ../../pom.xml
   
 
diff --git a/common/tags/pom.xml b/common/tags/pom.xml

[spark] branch branch-3.0 updated (60f4856 -> ed7be8c)

[ruifengz]

This is an automated email from the ASF dual-hosted git repository.

ruifengz pushed a change to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 60f4856  [SPARK-32701][CORE][DOCS] 
mapreduce.fileoutputcommitter.algorithm.version default value
 add 2b147c4  Preparing Spark release v3.0.1-rc3
 new ed7be8c  Preparing development version 3.0.2-SNAPSHOT

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] 01/01: Preparing Spark release v3.0.1-rc3

[ruifengz]

This is an automated email from the ASF dual-hosted git repository.

ruifengz pushed a commit to tag v3.0.1-rc3
in repository https://gitbox.apache.org/repos/asf/spark.git

commit 2b147c4cd50da32fe2b4167f97c8142102a0510d
Author: zhengruifeng 
AuthorDate: Fri Aug 28 03:25:14 2020 +

Preparing Spark release v3.0.1-rc3
---
 R/pkg/DESCRIPTION  | 2 +-
 assembly/pom.xml   | 2 +-
 common/kvstore/pom.xml | 2 +-
 common/network-common/pom.xml  | 2 +-
 common/network-shuffle/pom.xml | 2 +-
 common/network-yarn/pom.xml| 2 +-
 common/sketch/pom.xml  | 2 +-
 common/tags/pom.xml| 2 +-
 common/unsafe/pom.xml  | 2 +-
 core/pom.xml   | 2 +-
 docs/_config.yml   | 4 ++--
 examples/pom.xml   | 2 +-
 external/avro/pom.xml  | 2 +-
 external/docker-integration-tests/pom.xml  | 2 +-
 external/kafka-0-10-assembly/pom.xml   | 2 +-
 external/kafka-0-10-sql/pom.xml| 2 +-
 external/kafka-0-10-token-provider/pom.xml | 2 +-
 external/kafka-0-10/pom.xml| 2 +-
 external/kinesis-asl-assembly/pom.xml  | 2 +-
 external/kinesis-asl/pom.xml   | 2 +-
 external/spark-ganglia-lgpl/pom.xml| 2 +-
 graphx/pom.xml | 2 +-
 hadoop-cloud/pom.xml   | 2 +-
 launcher/pom.xml   | 2 +-
 mllib-local/pom.xml| 2 +-
 mllib/pom.xml  | 2 +-
 pom.xml| 2 +-
 python/pyspark/version.py  | 2 +-
 repl/pom.xml   | 2 +-
 resource-managers/kubernetes/core/pom.xml  | 2 +-
 resource-managers/kubernetes/integration-tests/pom.xml | 2 +-
 resource-managers/mesos/pom.xml| 2 +-
 resource-managers/yarn/pom.xml | 2 +-
 sql/catalyst/pom.xml   | 2 +-
 sql/core/pom.xml   | 2 +-
 sql/hive-thriftserver/pom.xml  | 2 +-
 sql/hive/pom.xml   | 2 +-
 streaming/pom.xml  | 2 +-
 tools/pom.xml  | 2 +-
 39 files changed, 40 insertions(+), 40 deletions(-)

diff --git a/R/pkg/DESCRIPTION b/R/pkg/DESCRIPTION
index a17dea9..86514f2 100644
--- a/R/pkg/DESCRIPTION
+++ b/R/pkg/DESCRIPTION
@@ -1,6 +1,6 @@
 Package: SparkR
 Type: Package
-Version: 3.0.2
+Version: 3.0.1
 Title: R Front End for 'Apache Spark'
 Description: Provides an R Front end for 'Apache Spark' 
.
 Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"),
diff --git a/assembly/pom.xml b/assembly/pom.xml
index e265bb9..c47425a 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -21,7 +21,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.2-SNAPSHOT
+3.0.1
 ../pom.xml
   
 
diff --git a/common/kvstore/pom.xml b/common/kvstore/pom.xml
index f433764..0948c6b 100644
--- a/common/kvstore/pom.xml
+++ b/common/kvstore/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.2-SNAPSHOT
+3.0.1
 ../../pom.xml
   
 
diff --git a/common/network-common/pom.xml b/common/network-common/pom.xml
index 4f31606..e7b16f1 100644
--- a/common/network-common/pom.xml
+++ b/common/network-common/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.2-SNAPSHOT
+3.0.1
 ../../pom.xml
   
 
diff --git a/common/network-shuffle/pom.xml b/common/network-shuffle/pom.xml
index a3ac661..c32a74f 100644
--- a/common/network-shuffle/pom.xml
+++ b/common/network-shuffle/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.2-SNAPSHOT
+3.0.1
 ../../pom.xml
   
 
diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml
index 5c5dfba..56ca725 100644
--- a/common/network-yarn/pom.xml
+++ b/common/network-yarn/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.2-SNAPSHOT
+3.0.1
 ../../pom.xml
   
 
diff --git a/common/sketch/pom.xml b/common/sketch/pom.xml
index 8041d7a..6b60cb3 100644
--- a/common/sketch/pom.xml
+++ b/common/sketch/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.2-SNAPSHOT
+3.0.1
 ../../pom.xml
   
 
diff --git a/common/tags/pom.xml b/common/tags/pom.xml
index 

[spark] tag v3.0.1-rc3 created (now 2b147c4)

[ruifengz]

This is an automated email from the ASF dual-hosted git repository.

ruifengz pushed a change to tag v3.0.1-rc3
in repository https://gitbox.apache.org/repos/asf/spark.git.


  at 2b147c4  (commit)
This tag includes the following new commits:

 new 2b147c4  Preparing Spark release v3.0.1-rc3

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (182727d -> d6c095c)

[yamamuro]

This is an automated email from the ASF dual-hosted git repository.

yamamuro pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 182727d  [SPARK-32713][K8S] Support execId placeholder in executor PVC 
conf
 add d6c095c  [SPARK-32693][SQL] Compare two dataframes with same schema 
except nullable property

No new revisions were added by this update.

Summary of changes:
 .../spark/sql/execution/joins/HashJoin.scala   |  7 ++--
 .../org/apache/spark/sql/DataFrameJoinSuite.scala  | 39 ++
 2 files changed, 44 insertions(+), 2 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (182727d -> d6c095c)

[yamamuro]

This is an automated email from the ASF dual-hosted git repository.

yamamuro pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 182727d  [SPARK-32713][K8S] Support execId placeholder in executor PVC 
conf
 add d6c095c  [SPARK-32693][SQL] Compare two dataframes with same schema 
except nullable property

No new revisions were added by this update.

Summary of changes:
 .../spark/sql/execution/joins/HashJoin.scala   |  7 ++--
 .../org/apache/spark/sql/DataFrameJoinSuite.scala  | 39 ++
 2 files changed, 44 insertions(+), 2 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (182727d -> d6c095c)

[yamamuro]

This is an automated email from the ASF dual-hosted git repository.

yamamuro pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 182727d  [SPARK-32713][K8S] Support execId placeholder in executor PVC 
conf
 add d6c095c  [SPARK-32693][SQL] Compare two dataframes with same schema 
except nullable property

No new revisions were added by this update.

Summary of changes:
 .../spark/sql/execution/joins/HashJoin.scala   |  7 ++--
 .../org/apache/spark/sql/DataFrameJoinSuite.scala  | 39 ++
 2 files changed, 44 insertions(+), 2 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (182727d -> d6c095c)

[yamamuro]

This is an automated email from the ASF dual-hosted git repository.

yamamuro pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 182727d  [SPARK-32713][K8S] Support execId placeholder in executor PVC 
conf
 add d6c095c  [SPARK-32693][SQL] Compare two dataframes with same schema 
except nullable property

No new revisions were added by this update.

Summary of changes:
 .../spark/sql/execution/joins/HashJoin.scala   |  7 ++--
 .../org/apache/spark/sql/DataFrameJoinSuite.scala  | 39 ++
 2 files changed, 44 insertions(+), 2 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (182727d -> d6c095c)

[yamamuro]

This is an automated email from the ASF dual-hosted git repository.

yamamuro pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 182727d  [SPARK-32713][K8S] Support execId placeholder in executor PVC 
conf
 add d6c095c  [SPARK-32693][SQL] Compare two dataframes with same schema 
except nullable property

No new revisions were added by this update.

Summary of changes:
 .../spark/sql/execution/joins/HashJoin.scala   |  7 ++--
 .../org/apache/spark/sql/DataFrameJoinSuite.scala  | 39 ++
 2 files changed, 44 insertions(+), 2 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[GitHub] [spark-website] srowen closed pull request #280: Add documentation for debugging with Maven and IntelliJ.

[GitBox]

srowen closed pull request #280:
URL: https://github.com/apache/spark-website/pull/280


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[GitHub] [spark-website] srowen closed pull request #287: Adds Kotlin to the list of third-party language bindings

[GitBox]

srowen closed pull request #287:
URL: https://github.com/apache/spark-website/pull/287


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark-website] branch asf-site updated: Adds Kotlin to the list of third-party language bindings

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/spark-website.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 36e19d7  Adds Kotlin to the list of third-party language bindings
36e19d7 is described below

commit 36e19d72f38a806b3086636ec1266bbffb8dfaf2
Author: MKhalusova 
AuthorDate: Thu Aug 27 17:47:46 2020 -0500

Adds Kotlin to the list of third-party language bindings

This PR adds a link to Kotlin for Apache Spark repo as a third-party 
language binding.

Author: MKhalusova 

Closes #287 from MKhalusova/kotlin-third-party.
---
 site/third-party-projects.html | 6 ++
 third-party-projects.md| 4 
 2 files changed, 10 insertions(+)

diff --git a/site/third-party-projects.html b/site/third-party-projects.html
index 5bd1524..bed5d61 100644
--- a/site/third-party-projects.html
+++ b/site/third-party-projects.html
@@ -298,6 +298,12 @@ transforming, and analyzing genomic data using Apache 
Spark
   https://github.com/dfdx/Spark.jl;>Spark.jl
 
 
+Kotlin
+
+
+  https://github.com/JetBrains/kotlin-spark-api;>Kotlin for 
Apache Spark
+
+
   
 
 
diff --git a/third-party-projects.md b/third-party-projects.md
index 6176ff6..8f29bbb 100644
--- a/third-party-projects.md
+++ b/third-party-projects.md
@@ -90,3 +90,7 @@ transforming, and analyzing genomic data using Apache Spark
 Julia
 
 - https://github.com/dfdx/Spark.jl;>Spark.jl
+
+Kotlin
+
+- https://github.com/JetBrains/kotlin-spark-api;>Kotlin for Apache 
Spark


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[GitHub] [spark-website] MKhalusova opened a new pull request #287: Adds Kotlin to the list of third-party language bindings

[GitBox]

MKhalusova opened a new pull request #287:
URL: https://github.com/apache/spark-website/pull/287


   This PR adds a link to Kotlin for Apache Spark repo as a third-party 
language binding. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (8749b2b -> 182727d)

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 8749b2b  [SPARK-32701][CORE][DOCS] 
mapreduce.fileoutputcommitter.algorithm.version default value
 add 182727d  [SPARK-32713][K8S] Support execId placeholder in executor PVC 
conf

No new revisions were added by this update.

Summary of changes:
 .../k8s/features/MountVolumesFeatureStep.scala |  9 +++-
 .../features/MountVolumesFeatureStepSuite.scala| 25 ++
 2 files changed, 33 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (8749b2b -> 182727d)

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 8749b2b  [SPARK-32701][CORE][DOCS] 
mapreduce.fileoutputcommitter.algorithm.version default value
 add 182727d  [SPARK-32713][K8S] Support execId placeholder in executor PVC 
conf

No new revisions were added by this update.

Summary of changes:
 .../k8s/features/MountVolumesFeatureStep.scala |  9 +++-
 .../features/MountVolumesFeatureStepSuite.scala| 25 ++
 2 files changed, 33 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (8749b2b -> 182727d)

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 8749b2b  [SPARK-32701][CORE][DOCS] 
mapreduce.fileoutputcommitter.algorithm.version default value
 add 182727d  [SPARK-32713][K8S] Support execId placeholder in executor PVC 
conf

No new revisions were added by this update.

Summary of changes:
 .../k8s/features/MountVolumesFeatureStep.scala |  9 +++-
 .../features/MountVolumesFeatureStepSuite.scala| 25 ++
 2 files changed, 33 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (8749b2b -> 182727d)

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 8749b2b  [SPARK-32701][CORE][DOCS] 
mapreduce.fileoutputcommitter.algorithm.version default value
 add 182727d  [SPARK-32713][K8S] Support execId placeholder in executor PVC 
conf

No new revisions were added by this update.

Summary of changes:
 .../k8s/features/MountVolumesFeatureStep.scala |  9 +++-
 .../features/MountVolumesFeatureStepSuite.scala| 25 ++
 2 files changed, 33 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (8749b2b -> 182727d)

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 8749b2b  [SPARK-32701][CORE][DOCS] 
mapreduce.fileoutputcommitter.algorithm.version default value
 add 182727d  [SPARK-32713][K8S] Support execId placeholder in executor PVC 
conf

No new revisions were added by this update.

Summary of changes:
 .../k8s/features/MountVolumesFeatureStep.scala |  9 +++-
 .../features/MountVolumesFeatureStepSuite.scala| 25 ++
 2 files changed, 33 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.0 updated: [SPARK-32701][CORE][DOCS] mapreduce.fileoutputcommitter.algorithm.version default value

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.0 by this push:
 new 60f4856  [SPARK-32701][CORE][DOCS] 
mapreduce.fileoutputcommitter.algorithm.version default value
60f4856 is described below

commit 60f485671a07a93ae8a8506ed2c0999cfe6ded7b
Author: waleedfateem 
AuthorDate: Thu Aug 27 09:05:50 2020 -0500

[SPARK-32701][CORE][DOCS] mapreduce.fileoutputcommitter.algorithm.version 
default value

The current documentation states that the default value of 
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version is 1 which is not 
entirely true since this configuration isn't set anywhere in Spark but rather 
inherited from the Hadoop FileOutputCommitter class.

### What changes were proposed in this pull request?

I'm submitting this change, to clarify that the default value will entirely 
depend on the Hadoop version of the runtime environment.

### Why are the changes needed?

An application would end up using algorithm version 1 on certain 
environments but without any changes the same exact application will use 
version 2 on environments running Hadoop 3.0 and later. This can have pretty 
bad consequences in certain scenarios, for example, two tasks can partially 
overwrite their output if speculation is enabled. Also, please refer to the 
following JIRA:
https://issues.apache.org/jira/browse/MAPREDUCE-7282

### Does this PR introduce _any_ user-facing change?

Yes. Configuration page content was modified where previously we explicitly 
highlighted that the default version for the FileOutputCommitter algorithm was 
v1, this now has changed to "Dependent on environment" with additional 
information in the description column to elaborate.

### How was this patch tested?

Checked changes locally in browser

Closes #29541 from waleedfateem/SPARK-32701.

Authored-by: waleedfateem 
Signed-off-by: Sean Owen 
(cherry picked from commit 8749b2b6fae5ee0ce7b48aae6d859ed71e98491d)
Signed-off-by: Sean Owen 
---
 docs/configuration.md | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/docs/configuration.md b/docs/configuration.md
index 2701fdb..95ff282 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1761,11 +1761,16 @@ Apart from these, the following properties are also 
available, and may be useful
 
 
   
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version
-  1
+  Dependent on environment
   
 The file output committer algorithm version, valid algorithm version 
number: 1 or 2.
 Version 2 may have better performance, but version 1 may handle failures 
better in certain situations,
 as per https://issues.apache.org/jira/browse/MAPREDUCE-4815;>MAPREDUCE-4815.
+The default value depends on the Hadoop version used in an environment:
+1 for Hadoop versions lower than 3.0
+2 for Hadoop versions 3.0 and higher
+It's important to note that this can change back to 1 again in the future 
once https://issues.apache.org/jira/browse/MAPREDUCE-7282;>MAPREDUCE-7282
+is fixed and merged.
   
   2.2.0
 


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (ed51a7f -> 8749b2b)

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from ed51a7f  [SPARK-30654] Bootstrap4 docs upgrade
 add 8749b2b  [SPARK-32701][CORE][DOCS] 
mapreduce.fileoutputcommitter.algorithm.version default value

No new revisions were added by this update.

Summary of changes:
 docs/configuration.md | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.0 updated: [SPARK-32701][CORE][DOCS] mapreduce.fileoutputcommitter.algorithm.version default value

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.0 by this push:
 new 60f4856  [SPARK-32701][CORE][DOCS] 
mapreduce.fileoutputcommitter.algorithm.version default value
60f4856 is described below

commit 60f485671a07a93ae8a8506ed2c0999cfe6ded7b
Author: waleedfateem 
AuthorDate: Thu Aug 27 09:05:50 2020 -0500

[SPARK-32701][CORE][DOCS] mapreduce.fileoutputcommitter.algorithm.version 
default value

The current documentation states that the default value of 
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version is 1 which is not 
entirely true since this configuration isn't set anywhere in Spark but rather 
inherited from the Hadoop FileOutputCommitter class.

### What changes were proposed in this pull request?

I'm submitting this change, to clarify that the default value will entirely 
depend on the Hadoop version of the runtime environment.

### Why are the changes needed?

An application would end up using algorithm version 1 on certain 
environments but without any changes the same exact application will use 
version 2 on environments running Hadoop 3.0 and later. This can have pretty 
bad consequences in certain scenarios, for example, two tasks can partially 
overwrite their output if speculation is enabled. Also, please refer to the 
following JIRA:
https://issues.apache.org/jira/browse/MAPREDUCE-7282

### Does this PR introduce _any_ user-facing change?

Yes. Configuration page content was modified where previously we explicitly 
highlighted that the default version for the FileOutputCommitter algorithm was 
v1, this now has changed to "Dependent on environment" with additional 
information in the description column to elaborate.

### How was this patch tested?

Checked changes locally in browser

Closes #29541 from waleedfateem/SPARK-32701.

Authored-by: waleedfateem 
Signed-off-by: Sean Owen 
(cherry picked from commit 8749b2b6fae5ee0ce7b48aae6d859ed71e98491d)
Signed-off-by: Sean Owen 
---
 docs/configuration.md | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/docs/configuration.md b/docs/configuration.md
index 2701fdb..95ff282 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1761,11 +1761,16 @@ Apart from these, the following properties are also 
available, and may be useful
 
 
   
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version
-  1
+  Dependent on environment
   
 The file output committer algorithm version, valid algorithm version 
number: 1 or 2.
 Version 2 may have better performance, but version 1 may handle failures 
better in certain situations,
 as per https://issues.apache.org/jira/browse/MAPREDUCE-4815;>MAPREDUCE-4815.
+The default value depends on the Hadoop version used in an environment:
+1 for Hadoop versions lower than 3.0
+2 for Hadoop versions 3.0 and higher
+It's important to note that this can change back to 1 again in the future 
once https://issues.apache.org/jira/browse/MAPREDUCE-7282;>MAPREDUCE-7282
+is fixed and merged.
   
   2.2.0
 


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (ed51a7f -> 8749b2b)

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from ed51a7f  [SPARK-30654] Bootstrap4 docs upgrade
 add 8749b2b  [SPARK-32701][CORE][DOCS] 
mapreduce.fileoutputcommitter.algorithm.version default value

No new revisions were added by this update.

Summary of changes:
 docs/configuration.md | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (f14f374 -> ed51a7f)

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from f14f374  [SPARK-32696][SQL][TEST-HIVE1.2][TEST-HADOOP2.7] Get columns 
operation should handle interval column properly
 add ed51a7f  [SPARK-30654] Bootstrap4 docs upgrade

No new revisions were added by this update.

Summary of changes:
 docs/_layouts/global.html  |  128 +-
 docs/css/bootstrap-responsive.css  | 1040 
 docs/css/bootstrap-responsive.min.css  |9 -
 docs/css/bootstrap.css | 5624 
 docs/css/bootstrap.min.css |   14 +-
 .../ui/static => docs/css}/bootstrap.min.css.map   |0
 docs/css/main.css  |  150 +-
 docs/js/main.js|   34 +-
 .../js/vendor}/bootstrap.bundle.min.js |0
 .../js/vendor}/bootstrap.bundle.min.js.map |0
 docs/js/vendor/bootstrap.js| 2027 ---
 docs/js/vendor/bootstrap.min.js|6 -
 12 files changed, 222 insertions(+), 8810 deletions(-)
 delete mode 100644 docs/css/bootstrap-responsive.css
 delete mode 100644 docs/css/bootstrap-responsive.min.css
 delete mode 100644 docs/css/bootstrap.css
 copy {core/src/main/resources/org/apache/spark/ui/static => 
docs/css}/bootstrap.min.css.map (100%)
 copy {core/src/main/resources/org/apache/spark/ui/static => 
docs/js/vendor}/bootstrap.bundle.min.js (100%)
 copy {core/src/main/resources/org/apache/spark/ui/static => 
docs/js/vendor}/bootstrap.bundle.min.js.map (100%)
 delete mode 100755 docs/js/vendor/bootstrap.js
 delete mode 100755 docs/js/vendor/bootstrap.min.js


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.0 updated: [SPARK-32701][CORE][DOCS] mapreduce.fileoutputcommitter.algorithm.version default value

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.0 by this push:
 new 60f4856  [SPARK-32701][CORE][DOCS] 
mapreduce.fileoutputcommitter.algorithm.version default value
60f4856 is described below

commit 60f485671a07a93ae8a8506ed2c0999cfe6ded7b
Author: waleedfateem 
AuthorDate: Thu Aug 27 09:05:50 2020 -0500

[SPARK-32701][CORE][DOCS] mapreduce.fileoutputcommitter.algorithm.version 
default value

The current documentation states that the default value of 
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version is 1 which is not 
entirely true since this configuration isn't set anywhere in Spark but rather 
inherited from the Hadoop FileOutputCommitter class.

### What changes were proposed in this pull request?

I'm submitting this change, to clarify that the default value will entirely 
depend on the Hadoop version of the runtime environment.

### Why are the changes needed?

An application would end up using algorithm version 1 on certain 
environments but without any changes the same exact application will use 
version 2 on environments running Hadoop 3.0 and later. This can have pretty 
bad consequences in certain scenarios, for example, two tasks can partially 
overwrite their output if speculation is enabled. Also, please refer to the 
following JIRA:
https://issues.apache.org/jira/browse/MAPREDUCE-7282

### Does this PR introduce _any_ user-facing change?

Yes. Configuration page content was modified where previously we explicitly 
highlighted that the default version for the FileOutputCommitter algorithm was 
v1, this now has changed to "Dependent on environment" with additional 
information in the description column to elaborate.

### How was this patch tested?

Checked changes locally in browser

Closes #29541 from waleedfateem/SPARK-32701.

Authored-by: waleedfateem 
Signed-off-by: Sean Owen 
(cherry picked from commit 8749b2b6fae5ee0ce7b48aae6d859ed71e98491d)
Signed-off-by: Sean Owen 
---
 docs/configuration.md | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/docs/configuration.md b/docs/configuration.md
index 2701fdb..95ff282 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1761,11 +1761,16 @@ Apart from these, the following properties are also 
available, and may be useful
 
 
   
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version
-  1
+  Dependent on environment
   
 The file output committer algorithm version, valid algorithm version 
number: 1 or 2.
 Version 2 may have better performance, but version 1 may handle failures 
better in certain situations,
 as per https://issues.apache.org/jira/browse/MAPREDUCE-4815;>MAPREDUCE-4815.
+The default value depends on the Hadoop version used in an environment:
+1 for Hadoop versions lower than 3.0
+2 for Hadoop versions 3.0 and higher
+It's important to note that this can change back to 1 again in the future 
once https://issues.apache.org/jira/browse/MAPREDUCE-7282;>MAPREDUCE-7282
+is fixed and merged.
   
   2.2.0
 


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (ed51a7f -> 8749b2b)

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from ed51a7f  [SPARK-30654] Bootstrap4 docs upgrade
 add 8749b2b  [SPARK-32701][CORE][DOCS] 
mapreduce.fileoutputcommitter.algorithm.version default value

No new revisions were added by this update.

Summary of changes:
 docs/configuration.md | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (f14f374 -> ed51a7f)

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from f14f374  [SPARK-32696][SQL][TEST-HIVE1.2][TEST-HADOOP2.7] Get columns 
operation should handle interval column properly
 add ed51a7f  [SPARK-30654] Bootstrap4 docs upgrade

No new revisions were added by this update.

Summary of changes:
 docs/_layouts/global.html  |  128 +-
 docs/css/bootstrap-responsive.css  | 1040 
 docs/css/bootstrap-responsive.min.css  |9 -
 docs/css/bootstrap.css | 5624 
 docs/css/bootstrap.min.css |   14 +-
 .../ui/static => docs/css}/bootstrap.min.css.map   |0
 docs/css/main.css  |  150 +-
 docs/js/main.js|   34 +-
 .../js/vendor}/bootstrap.bundle.min.js |0
 .../js/vendor}/bootstrap.bundle.min.js.map |0
 docs/js/vendor/bootstrap.js| 2027 ---
 docs/js/vendor/bootstrap.min.js|6 -
 12 files changed, 222 insertions(+), 8810 deletions(-)
 delete mode 100644 docs/css/bootstrap-responsive.css
 delete mode 100644 docs/css/bootstrap-responsive.min.css
 delete mode 100644 docs/css/bootstrap.css
 copy {core/src/main/resources/org/apache/spark/ui/static => 
docs/css}/bootstrap.min.css.map (100%)
 copy {core/src/main/resources/org/apache/spark/ui/static => 
docs/js/vendor}/bootstrap.bundle.min.js (100%)
 copy {core/src/main/resources/org/apache/spark/ui/static => 
docs/js/vendor}/bootstrap.bundle.min.js.map (100%)
 delete mode 100755 docs/js/vendor/bootstrap.js
 delete mode 100755 docs/js/vendor/bootstrap.min.js


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.0 updated: [SPARK-32701][CORE][DOCS] mapreduce.fileoutputcommitter.algorithm.version default value

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.0 by this push:
 new 60f4856  [SPARK-32701][CORE][DOCS] 
mapreduce.fileoutputcommitter.algorithm.version default value
60f4856 is described below

commit 60f485671a07a93ae8a8506ed2c0999cfe6ded7b
Author: waleedfateem 
AuthorDate: Thu Aug 27 09:05:50 2020 -0500

[SPARK-32701][CORE][DOCS] mapreduce.fileoutputcommitter.algorithm.version 
default value

The current documentation states that the default value of 
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version is 1 which is not 
entirely true since this configuration isn't set anywhere in Spark but rather 
inherited from the Hadoop FileOutputCommitter class.

### What changes were proposed in this pull request?

I'm submitting this change, to clarify that the default value will entirely 
depend on the Hadoop version of the runtime environment.

### Why are the changes needed?

An application would end up using algorithm version 1 on certain 
environments but without any changes the same exact application will use 
version 2 on environments running Hadoop 3.0 and later. This can have pretty 
bad consequences in certain scenarios, for example, two tasks can partially 
overwrite their output if speculation is enabled. Also, please refer to the 
following JIRA:
https://issues.apache.org/jira/browse/MAPREDUCE-7282

### Does this PR introduce _any_ user-facing change?

Yes. Configuration page content was modified where previously we explicitly 
highlighted that the default version for the FileOutputCommitter algorithm was 
v1, this now has changed to "Dependent on environment" with additional 
information in the description column to elaborate.

### How was this patch tested?

Checked changes locally in browser

Closes #29541 from waleedfateem/SPARK-32701.

Authored-by: waleedfateem 
Signed-off-by: Sean Owen 
(cherry picked from commit 8749b2b6fae5ee0ce7b48aae6d859ed71e98491d)
Signed-off-by: Sean Owen 
---
 docs/configuration.md | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/docs/configuration.md b/docs/configuration.md
index 2701fdb..95ff282 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1761,11 +1761,16 @@ Apart from these, the following properties are also 
available, and may be useful
 
 
   
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version
-  1
+  Dependent on environment
   
 The file output committer algorithm version, valid algorithm version 
number: 1 or 2.
 Version 2 may have better performance, but version 1 may handle failures 
better in certain situations,
 as per https://issues.apache.org/jira/browse/MAPREDUCE-4815;>MAPREDUCE-4815.
+The default value depends on the Hadoop version used in an environment:
+1 for Hadoop versions lower than 3.0
+2 for Hadoop versions 3.0 and higher
+It's important to note that this can change back to 1 again in the future 
once https://issues.apache.org/jira/browse/MAPREDUCE-7282;>MAPREDUCE-7282
+is fixed and merged.
   
   2.2.0
 


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (ed51a7f -> 8749b2b)

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from ed51a7f  [SPARK-30654] Bootstrap4 docs upgrade
 add 8749b2b  [SPARK-32701][CORE][DOCS] 
mapreduce.fileoutputcommitter.algorithm.version default value

No new revisions were added by this update.

Summary of changes:
 docs/configuration.md | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (f14f374 -> ed51a7f)

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from f14f374  [SPARK-32696][SQL][TEST-HIVE1.2][TEST-HADOOP2.7] Get columns 
operation should handle interval column properly
 add ed51a7f  [SPARK-30654] Bootstrap4 docs upgrade

No new revisions were added by this update.

Summary of changes:
 docs/_layouts/global.html  |  128 +-
 docs/css/bootstrap-responsive.css  | 1040 
 docs/css/bootstrap-responsive.min.css  |9 -
 docs/css/bootstrap.css | 5624 
 docs/css/bootstrap.min.css |   14 +-
 .../ui/static => docs/css}/bootstrap.min.css.map   |0
 docs/css/main.css  |  150 +-
 docs/js/main.js|   34 +-
 .../js/vendor}/bootstrap.bundle.min.js |0
 .../js/vendor}/bootstrap.bundle.min.js.map |0
 docs/js/vendor/bootstrap.js| 2027 ---
 docs/js/vendor/bootstrap.min.js|6 -
 12 files changed, 222 insertions(+), 8810 deletions(-)
 delete mode 100644 docs/css/bootstrap-responsive.css
 delete mode 100644 docs/css/bootstrap-responsive.min.css
 delete mode 100644 docs/css/bootstrap.css
 copy {core/src/main/resources/org/apache/spark/ui/static => 
docs/css}/bootstrap.min.css.map (100%)
 copy {core/src/main/resources/org/apache/spark/ui/static => 
docs/js/vendor}/bootstrap.bundle.min.js (100%)
 copy {core/src/main/resources/org/apache/spark/ui/static => 
docs/js/vendor}/bootstrap.bundle.min.js.map (100%)
 delete mode 100755 docs/js/vendor/bootstrap.js
 delete mode 100755 docs/js/vendor/bootstrap.min.js


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.0 updated: [SPARK-32701][CORE][DOCS] mapreduce.fileoutputcommitter.algorithm.version default value

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.0 by this push:
 new 60f4856  [SPARK-32701][CORE][DOCS] 
mapreduce.fileoutputcommitter.algorithm.version default value
60f4856 is described below

commit 60f485671a07a93ae8a8506ed2c0999cfe6ded7b
Author: waleedfateem 
AuthorDate: Thu Aug 27 09:05:50 2020 -0500

[SPARK-32701][CORE][DOCS] mapreduce.fileoutputcommitter.algorithm.version 
default value

The current documentation states that the default value of 
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version is 1 which is not 
entirely true since this configuration isn't set anywhere in Spark but rather 
inherited from the Hadoop FileOutputCommitter class.

### What changes were proposed in this pull request?

I'm submitting this change, to clarify that the default value will entirely 
depend on the Hadoop version of the runtime environment.

### Why are the changes needed?

An application would end up using algorithm version 1 on certain 
environments but without any changes the same exact application will use 
version 2 on environments running Hadoop 3.0 and later. This can have pretty 
bad consequences in certain scenarios, for example, two tasks can partially 
overwrite their output if speculation is enabled. Also, please refer to the 
following JIRA:
https://issues.apache.org/jira/browse/MAPREDUCE-7282

### Does this PR introduce _any_ user-facing change?

Yes. Configuration page content was modified where previously we explicitly 
highlighted that the default version for the FileOutputCommitter algorithm was 
v1, this now has changed to "Dependent on environment" with additional 
information in the description column to elaborate.

### How was this patch tested?

Checked changes locally in browser

Closes #29541 from waleedfateem/SPARK-32701.

Authored-by: waleedfateem 
Signed-off-by: Sean Owen 
(cherry picked from commit 8749b2b6fae5ee0ce7b48aae6d859ed71e98491d)
Signed-off-by: Sean Owen 
---
 docs/configuration.md | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/docs/configuration.md b/docs/configuration.md
index 2701fdb..95ff282 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1761,11 +1761,16 @@ Apart from these, the following properties are also 
available, and may be useful
 
 
   
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version
-  1
+  Dependent on environment
   
 The file output committer algorithm version, valid algorithm version 
number: 1 or 2.
 Version 2 may have better performance, but version 1 may handle failures 
better in certain situations,
 as per https://issues.apache.org/jira/browse/MAPREDUCE-4815;>MAPREDUCE-4815.
+The default value depends on the Hadoop version used in an environment:
+1 for Hadoop versions lower than 3.0
+2 for Hadoop versions 3.0 and higher
+It's important to note that this can change back to 1 again in the future 
once https://issues.apache.org/jira/browse/MAPREDUCE-7282;>MAPREDUCE-7282
+is fixed and merged.
   
   2.2.0
 


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (ed51a7f -> 8749b2b)

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from ed51a7f  [SPARK-30654] Bootstrap4 docs upgrade
 add 8749b2b  [SPARK-32701][CORE][DOCS] 
mapreduce.fileoutputcommitter.algorithm.version default value

No new revisions were added by this update.

Summary of changes:
 docs/configuration.md | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (f14f374 -> ed51a7f)

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from f14f374  [SPARK-32696][SQL][TEST-HIVE1.2][TEST-HADOOP2.7] Get columns 
operation should handle interval column properly
 add ed51a7f  [SPARK-30654] Bootstrap4 docs upgrade

No new revisions were added by this update.

Summary of changes:
 docs/_layouts/global.html  |  128 +-
 docs/css/bootstrap-responsive.css  | 1040 
 docs/css/bootstrap-responsive.min.css  |9 -
 docs/css/bootstrap.css | 5624 
 docs/css/bootstrap.min.css |   14 +-
 .../ui/static => docs/css}/bootstrap.min.css.map   |0
 docs/css/main.css  |  150 +-
 docs/js/main.js|   34 +-
 .../js/vendor}/bootstrap.bundle.min.js |0
 .../js/vendor}/bootstrap.bundle.min.js.map |0
 docs/js/vendor/bootstrap.js| 2027 ---
 docs/js/vendor/bootstrap.min.js|6 -
 12 files changed, 222 insertions(+), 8810 deletions(-)
 delete mode 100644 docs/css/bootstrap-responsive.css
 delete mode 100644 docs/css/bootstrap-responsive.min.css
 delete mode 100644 docs/css/bootstrap.css
 copy {core/src/main/resources/org/apache/spark/ui/static => 
docs/css}/bootstrap.min.css.map (100%)
 copy {core/src/main/resources/org/apache/spark/ui/static => 
docs/js/vendor}/bootstrap.bundle.min.js (100%)
 copy {core/src/main/resources/org/apache/spark/ui/static => 
docs/js/vendor}/bootstrap.bundle.min.js.map (100%)
 delete mode 100755 docs/js/vendor/bootstrap.js
 delete mode 100755 docs/js/vendor/bootstrap.min.js


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (f14f374 -> ed51a7f)

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from f14f374  [SPARK-32696][SQL][TEST-HIVE1.2][TEST-HADOOP2.7] Get columns 
operation should handle interval column properly
 add ed51a7f  [SPARK-30654] Bootstrap4 docs upgrade

No new revisions were added by this update.

Summary of changes:
 docs/_layouts/global.html  |  128 +-
 docs/css/bootstrap-responsive.css  | 1040 
 docs/css/bootstrap-responsive.min.css  |9 -
 docs/css/bootstrap.css | 5624 
 docs/css/bootstrap.min.css |   14 +-
 .../ui/static => docs/css}/bootstrap.min.css.map   |0
 docs/css/main.css  |  150 +-
 docs/js/main.js|   34 +-
 .../js/vendor}/bootstrap.bundle.min.js |0
 .../js/vendor}/bootstrap.bundle.min.js.map |0
 docs/js/vendor/bootstrap.js| 2027 ---
 docs/js/vendor/bootstrap.min.js|6 -
 12 files changed, 222 insertions(+), 8810 deletions(-)
 delete mode 100644 docs/css/bootstrap-responsive.css
 delete mode 100644 docs/css/bootstrap-responsive.min.css
 delete mode 100644 docs/css/bootstrap.css
 copy {core/src/main/resources/org/apache/spark/ui/static => 
docs/css}/bootstrap.min.css.map (100%)
 copy {core/src/main/resources/org/apache/spark/ui/static => 
docs/js/vendor}/bootstrap.bundle.min.js (100%)
 copy {core/src/main/resources/org/apache/spark/ui/static => 
docs/js/vendor}/bootstrap.bundle.min.js.map (100%)
 delete mode 100755 docs/js/vendor/bootstrap.js
 delete mode 100755 docs/js/vendor/bootstrap.min.js


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (eb37976 -> f14f374)

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from eb37976  [SPARK-32705][SQL] Fix serialization issue for 
EmptyHashedRelation
 add f14f374  [SPARK-32696][SQL][TEST-HIVE1.2][TEST-HADOOP2.7] Get columns 
operation should handle interval column properly

No new revisions were added by this update.

Summary of changes:
 .../thriftserver/SparkGetColumnsOperation.scala| 28 ++---
 .../thriftserver/HiveThriftServer2Suites.scala | 17 +
 .../thriftserver/SparkMetadataOperationSuite.scala | 29 +-
 .../hive/thriftserver/ThriftserverShimUtils.scala  |  2 --
 .../hive/thriftserver/ThriftserverShimUtils.scala  |  2 --
 5 files changed, 70 insertions(+), 8 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (eb37976 -> f14f374)

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from eb37976  [SPARK-32705][SQL] Fix serialization issue for 
EmptyHashedRelation
 add f14f374  [SPARK-32696][SQL][TEST-HIVE1.2][TEST-HADOOP2.7] Get columns 
operation should handle interval column properly

No new revisions were added by this update.

Summary of changes:
 .../thriftserver/SparkGetColumnsOperation.scala| 28 ++---
 .../thriftserver/HiveThriftServer2Suites.scala | 17 +
 .../thriftserver/SparkMetadataOperationSuite.scala | 29 +-
 .../hive/thriftserver/ThriftserverShimUtils.scala  |  2 --
 .../hive/thriftserver/ThriftserverShimUtils.scala  |  2 --
 5 files changed, 70 insertions(+), 8 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (eb37976 -> f14f374)

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from eb37976  [SPARK-32705][SQL] Fix serialization issue for 
EmptyHashedRelation
 add f14f374  [SPARK-32696][SQL][TEST-HIVE1.2][TEST-HADOOP2.7] Get columns 
operation should handle interval column properly

No new revisions were added by this update.

Summary of changes:
 .../thriftserver/SparkGetColumnsOperation.scala| 28 ++---
 .../thriftserver/HiveThriftServer2Suites.scala | 17 +
 .../thriftserver/SparkMetadataOperationSuite.scala | 29 +-
 .../hive/thriftserver/ThriftserverShimUtils.scala  |  2 --
 .../hive/thriftserver/ThriftserverShimUtils.scala  |  2 --
 5 files changed, 70 insertions(+), 8 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (eb37976 -> f14f374)

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from eb37976  [SPARK-32705][SQL] Fix serialization issue for 
EmptyHashedRelation
 add f14f374  [SPARK-32696][SQL][TEST-HIVE1.2][TEST-HADOOP2.7] Get columns 
operation should handle interval column properly

No new revisions were added by this update.

Summary of changes:
 .../thriftserver/SparkGetColumnsOperation.scala| 28 ++---
 .../thriftserver/HiveThriftServer2Suites.scala | 17 +
 .../thriftserver/SparkMetadataOperationSuite.scala | 29 +-
 .../hive/thriftserver/ThriftserverShimUtils.scala  |  2 --
 .../hive/thriftserver/ThriftserverShimUtils.scala  |  2 --
 5 files changed, 70 insertions(+), 8 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (eb37976 -> f14f374)

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from eb37976  [SPARK-32705][SQL] Fix serialization issue for 
EmptyHashedRelation
 add f14f374  [SPARK-32696][SQL][TEST-HIVE1.2][TEST-HADOOP2.7] Get columns 
operation should handle interval column properly

No new revisions were added by this update.

Summary of changes:
 .../thriftserver/SparkGetColumnsOperation.scala| 28 ++---
 .../thriftserver/HiveThriftServer2Suites.scala | 17 +
 .../thriftserver/SparkMetadataOperationSuite.scala | 29 +-
 .../hive/thriftserver/ThriftserverShimUtils.scala  |  2 --
 .../hive/thriftserver/ThriftserverShimUtils.scala  |  2 --
 5 files changed, 70 insertions(+), 8 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (baaa756 -> eb37976)

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from baaa756  [SPARK-32516][SQL][FOLLOWUP] 'path' option cannot coexist 
with path parameter for DataFrameWriter.save(), DataStreamReader.load() and 
DataStreamWriter.start()
 add eb37976  [SPARK-32705][SQL] Fix serialization issue for 
EmptyHashedRelation

No new revisions were added by this update.

Summary of changes:
 .../spark/sql/execution/joins/HashedRelation.scala | 85 ++
 1 file changed, 38 insertions(+), 47 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (baaa756 -> eb37976)

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from baaa756  [SPARK-32516][SQL][FOLLOWUP] 'path' option cannot coexist 
with path parameter for DataFrameWriter.save(), DataStreamReader.load() and 
DataStreamWriter.start()
 add eb37976  [SPARK-32705][SQL] Fix serialization issue for 
EmptyHashedRelation

No new revisions were added by this update.

Summary of changes:
 .../spark/sql/execution/joins/HashedRelation.scala | 85 ++
 1 file changed, 38 insertions(+), 47 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (b786f31 -> baaa756)

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from b786f31  [SPARK-32643][CORE][K8S] Consolidate state decommissioning in 
the TaskSchedulerImpl realm
 add baaa756  [SPARK-32516][SQL][FOLLOWUP] 'path' option cannot coexist 
with path parameter for DataFrameWriter.save(), DataStreamReader.load() and 
DataStreamWriter.start()

No new revisions were added by this update.

Summary of changes:
 docs/sql-migration-guide.md|  2 +-
 python/pyspark/sql/tests/test_streaming.py | 19 +---
 .../org/apache/spark/sql/internal/SQLConf.scala|  5 ++-
 .../org/apache/spark/sql/DataFrameReader.scala |  3 +-
 .../org/apache/spark/sql/DataFrameWriter.scala |  7 +++
 .../spark/sql/streaming/DataStreamReader.scala |  7 +++
 .../spark/sql/streaming/DataStreamWriter.scala |  7 +++
 .../test/DataStreamReaderWriterSuite.scala | 51 +-
 .../sql/test/DataFrameReaderWriterSuite.scala  | 33 --
 9 files changed, 119 insertions(+), 15 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (baaa756 -> eb37976)

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from baaa756  [SPARK-32516][SQL][FOLLOWUP] 'path' option cannot coexist 
with path parameter for DataFrameWriter.save(), DataStreamReader.load() and 
DataStreamWriter.start()
 add eb37976  [SPARK-32705][SQL] Fix serialization issue for 
EmptyHashedRelation

No new revisions were added by this update.

Summary of changes:
 .../spark/sql/execution/joins/HashedRelation.scala | 85 ++
 1 file changed, 38 insertions(+), 47 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (b786f31 -> baaa756)

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from b786f31  [SPARK-32643][CORE][K8S] Consolidate state decommissioning in 
the TaskSchedulerImpl realm
 add baaa756  [SPARK-32516][SQL][FOLLOWUP] 'path' option cannot coexist 
with path parameter for DataFrameWriter.save(), DataStreamReader.load() and 
DataStreamWriter.start()

No new revisions were added by this update.

Summary of changes:
 docs/sql-migration-guide.md|  2 +-
 python/pyspark/sql/tests/test_streaming.py | 19 +---
 .../org/apache/spark/sql/internal/SQLConf.scala|  5 ++-
 .../org/apache/spark/sql/DataFrameReader.scala |  3 +-
 .../org/apache/spark/sql/DataFrameWriter.scala |  7 +++
 .../spark/sql/streaming/DataStreamReader.scala |  7 +++
 .../spark/sql/streaming/DataStreamWriter.scala |  7 +++
 .../test/DataStreamReaderWriterSuite.scala | 51 +-
 .../sql/test/DataFrameReaderWriterSuite.scala  | 33 --
 9 files changed, 119 insertions(+), 15 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (baaa756 -> eb37976)

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from baaa756  [SPARK-32516][SQL][FOLLOWUP] 'path' option cannot coexist 
with path parameter for DataFrameWriter.save(), DataStreamReader.load() and 
DataStreamWriter.start()
 add eb37976  [SPARK-32705][SQL] Fix serialization issue for 
EmptyHashedRelation

No new revisions were added by this update.

Summary of changes:
 .../spark/sql/execution/joins/HashedRelation.scala | 85 ++
 1 file changed, 38 insertions(+), 47 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (b786f31 -> baaa756)

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from b786f31  [SPARK-32643][CORE][K8S] Consolidate state decommissioning in 
the TaskSchedulerImpl realm
 add baaa756  [SPARK-32516][SQL][FOLLOWUP] 'path' option cannot coexist 
with path parameter for DataFrameWriter.save(), DataStreamReader.load() and 
DataStreamWriter.start()

No new revisions were added by this update.

Summary of changes:
 docs/sql-migration-guide.md|  2 +-
 python/pyspark/sql/tests/test_streaming.py | 19 +---
 .../org/apache/spark/sql/internal/SQLConf.scala|  5 ++-
 .../org/apache/spark/sql/DataFrameReader.scala |  3 +-
 .../org/apache/spark/sql/DataFrameWriter.scala |  7 +++
 .../spark/sql/streaming/DataStreamReader.scala |  7 +++
 .../spark/sql/streaming/DataStreamWriter.scala |  7 +++
 .../test/DataStreamReaderWriterSuite.scala | 51 +-
 .../sql/test/DataFrameReaderWriterSuite.scala  | 33 --
 9 files changed, 119 insertions(+), 15 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (baaa756 -> eb37976)

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from baaa756  [SPARK-32516][SQL][FOLLOWUP] 'path' option cannot coexist 
with path parameter for DataFrameWriter.save(), DataStreamReader.load() and 
DataStreamWriter.start()
 add eb37976  [SPARK-32705][SQL] Fix serialization issue for 
EmptyHashedRelation

No new revisions were added by this update.

Summary of changes:
 .../spark/sql/execution/joins/HashedRelation.scala | 85 ++
 1 file changed, 38 insertions(+), 47 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (b786f31 -> baaa756)

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from b786f31  [SPARK-32643][CORE][K8S] Consolidate state decommissioning in 
the TaskSchedulerImpl realm
 add baaa756  [SPARK-32516][SQL][FOLLOWUP] 'path' option cannot coexist 
with path parameter for DataFrameWriter.save(), DataStreamReader.load() and 
DataStreamWriter.start()

No new revisions were added by this update.

Summary of changes:
 docs/sql-migration-guide.md|  2 +-
 python/pyspark/sql/tests/test_streaming.py | 19 +---
 .../org/apache/spark/sql/internal/SQLConf.scala|  5 ++-
 .../org/apache/spark/sql/DataFrameReader.scala |  3 +-
 .../org/apache/spark/sql/DataFrameWriter.scala |  7 +++
 .../spark/sql/streaming/DataStreamReader.scala |  7 +++
 .../spark/sql/streaming/DataStreamWriter.scala |  7 +++
 .../test/DataStreamReaderWriterSuite.scala | 51 +-
 .../sql/test/DataFrameReaderWriterSuite.scala  | 33 --
 9 files changed, 119 insertions(+), 15 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (b786f31 -> baaa756)

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from b786f31  [SPARK-32643][CORE][K8S] Consolidate state decommissioning in 
the TaskSchedulerImpl realm
 add baaa756  [SPARK-32516][SQL][FOLLOWUP] 'path' option cannot coexist 
with path parameter for DataFrameWriter.save(), DataStreamReader.load() and 
DataStreamWriter.start()

No new revisions were added by this update.

Summary of changes:
 docs/sql-migration-guide.md|  2 +-
 python/pyspark/sql/tests/test_streaming.py | 19 +---
 .../org/apache/spark/sql/internal/SQLConf.scala|  5 ++-
 .../org/apache/spark/sql/DataFrameReader.scala |  3 +-
 .../org/apache/spark/sql/DataFrameWriter.scala |  7 +++
 .../spark/sql/streaming/DataStreamReader.scala |  7 +++
 .../spark/sql/streaming/DataStreamWriter.scala |  7 +++
 .../test/DataStreamReaderWriterSuite.scala | 51 +-
 .../sql/test/DataFrameReaderWriterSuite.scala  | 33 --
 9 files changed, 119 insertions(+), 15 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org