spark git commit: [SPARK-23303][SQL] improve the explain result for data source v2 relations

Repository: spark Updated Branches: refs/heads/master 8c5b34c42 -> ad640a5af [SPARK-23303][SQL] improve the explain result for data source v2 relations ## What changes were proposed in this pull request? The proposed explain format: **[streaming header] [RelationV2/ScanV2] [data source name]

svn commit: r25476 - in /dev/spark/2.4.0-SNAPSHOT-2018_03_05_20_01-8c5b34c-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Tue Mar 6 04:15:37 2018 New Revision: 25476 Log: Apache Spark 2.4.0-SNAPSHOT-2018_03_05_20_01-8c5b34c docs [This commit notification would consist of 1444 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r25471 - in /dev/spark/2.3.1-SNAPSHOT-2018_03_05_18_01-b9ea2e8-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Tue Mar 6 02:15:45 2018 New Revision: 25471 Log: Apache Spark 2.3.1-SNAPSHOT-2018_03_05_18_01-b9ea2e8 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23604][SQL] Change Statistics.isEmpty to !Statistics.hasNonNul…

Repository: spark Updated Branches: refs/heads/master f6b49f9d1 -> 8c5b34c42 [SPARK-23604][SQL] Change Statistics.isEmpty to !Statistics.hasNonNul… …lValue ## What changes were proposed in this pull request? Parquet 1.9 will change the semantics of Statistics.isEmpty slightly to reflect

spark git commit: [SPARK-23586][SQL] Add interpreted execution to WrapOption

Repository: spark Updated Branches: refs/heads/master 7706eea6a -> f6b49f9d1 [SPARK-23586][SQL] Add interpreted execution to WrapOption ## What changes were proposed in this pull request? The PR adds interpreted execution to WrapOption. ## How was this patch tested? added UT Author: Marco

svn commit: r25462 - in /dev/spark/2.4.0-SNAPSHOT-2018_03_05_16_01-7706eea-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Tue Mar 6 00:15:31 2018 New Revision: 25462 Log: Apache Spark 2.4.0-SNAPSHOT-2018_03_05_16_01-7706eea docs [This commit notification would consist of 1444 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-18630][PYTHON][ML] Move del method from JavaParams to JavaWrapper; add tests

Repository: spark Updated Branches: refs/heads/master 508573958 -> 7706eea6a [SPARK-18630][PYTHON][ML] Move del method from JavaParams to JavaWrapper; add tests The `__del__` method that explicitly detaches the object was moved from `JavaParams` to `JavaWrapper` class, this way model

spark git commit: [SPARK-23538][CORE] Remove custom configuration for SSL client.

Repository: spark Updated Branches: refs/heads/master f2cab56ca -> 508573958 [SPARK-23538][CORE] Remove custom configuration for SSL client. These options were used to configure the built-in JRE SSL libraries when downloading files from HTTPS servers. But because they were also used to set up

spark git commit: [SPARK-23040][CORE] Returns interruptible iterator for shuffle reader

Repository: spark Updated Branches: refs/heads/master b0f422c38 -> f2cab56ca [SPARK-23040][CORE] Returns interruptible iterator for shuffle reader ## What changes were proposed in this pull request? Before this commit, a non-interruptible iterator is returned if aggregator or ordering is

spark git commit: [SPARK-23434][SQL][BRANCH-2.2] Spark should not warn `metadata directory` for a HDFS file path

Repository: spark Updated Branches: refs/heads/branch-2.2 9bd25c9bf -> 4864d2104 [SPARK-23434][SQL][BRANCH-2.2] Spark should not warn `metadata directory` for a HDFS file path ## What changes were proposed in this pull request? In a kerberized cluster, when Spark reads a file path (e.g.

spark git commit: [SPARK-23434][SQL][BRANCH-2.3] Spark should not warn `metadata directory` for a HDFS file path

Repository: spark Updated Branches: refs/heads/branch-2.3 911b83da4 -> b9ea2e87b [SPARK-23434][SQL][BRANCH-2.3] Spark should not warn `metadata directory` for a HDFS file path ## What changes were proposed in this pull request? In a kerberized cluster, when Spark reads a file path (e.g.

svn commit: r25458 - in /dev/spark/2.3.1-SNAPSHOT-2018_03_05_14_01-911b83d-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Mon Mar 5 22:15:15 2018 New Revision: 25458 Log: Apache Spark 2.3.1-SNAPSHOT-2018_03_05_14_01-911b83d docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23559][SS] Add epoch ID to DataWriterFactory.

Repository: spark Updated Branches: refs/heads/master ba622f45c -> b0f422c38 [SPARK-23559][SS] Add epoch ID to DataWriterFactory. ## What changes were proposed in this pull request? Add an epoch ID argument to DataWriterFactory for use in streaming. As a side effect of passing in this

svn commit: r25453 - in /dev/spark/2.4.0-SNAPSHOT-2018_03_05_12_01-ba622f4-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Mon Mar 5 20:20:52 2018 New Revision: 25453 Log: Apache Spark 2.4.0-SNAPSHOT-2018_03_05_12_01-ba622f4 docs [This commit notification would consist of 1444 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23457][SQL][BRANCH-2.3] Register task completion listeners first in ParquetFileFormat

Repository: spark Updated Branches: refs/heads/branch-2.3 4550673b1 -> 911b83da4 [SPARK-23457][SQL][BRANCH-2.3] Register task completion listeners first in ParquetFileFormat ## What changes were proposed in this pull request? ParquetFileFormat leaks opened files in some cases. This PR

spark git commit: [SPARK-23585][SQL] Add interpreted execution to UnwrapOption

Repository: spark Updated Branches: refs/heads/master 98a5c0a35 -> ba622f45c [SPARK-23585][SQL] Add interpreted execution to UnwrapOption ## What changes were proposed in this pull request? The PR adds interpreted execution to UnwrapOption. ## How was this patch tested? added UT Author:

spark git commit: [SPARK-22882][ML][TESTS] ML test for structured streaming: ml.classification

Repository: spark Updated Branches: refs/heads/master 4586eada4 -> 98a5c0a35 [SPARK-22882][ML][TESTS] ML test for structured streaming: ml.classification ## What changes were proposed in this pull request? adding Structured Streaming tests for all Models/Transformers in

spark git commit: [SPARK-22882][ML][TESTS] ML test for structured streaming: ml.classification

Repository: spark Updated Branches: refs/heads/branch-2.3 232b9f81f -> 4550673b1 [SPARK-22882][ML][TESTS] ML test for structured streaming: ml.classification ## What changes were proposed in this pull request? adding Structured Streaming tests for all Models/Transformers in

svn commit: r25445 - in /dev/spark/2.3.1-SNAPSHOT-2018_03_05_10_01-232b9f8-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Mon Mar 5 18:16:47 2018 New Revision: 25445 Log: Apache Spark 2.3.1-SNAPSHOT-2018_03_05_10_01-232b9f8 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

[2/3] spark git commit: [SPARK-22430][R][DOCS] Unknown tag warnings when building R docs with Roxygen 6.0.1

http://git-wip-us.apache.org/repos/asf/spark/blob/4586eada/R/pkg/R/generics.R -- diff --git a/R/pkg/R/generics.R b/R/pkg/R/generics.R index e0dde33..6fba4b6 100644 --- a/R/pkg/R/generics.R +++ b/R/pkg/R/generics.R @@ -19,7 +19,6

[1/3] spark git commit: [SPARK-22430][R][DOCS] Unknown tag warnings when building R docs with Roxygen 6.0.1

Repository: spark Updated Branches: refs/heads/master 947b4e6f0 -> 4586eada4 http://git-wip-us.apache.org/repos/asf/spark/blob/4586eada/R/pkg/R/mllib_tree.R -- diff --git a/R/pkg/R/mllib_tree.R b/R/pkg/R/mllib_tree.R index

[3/3] spark git commit: [SPARK-22430][R][DOCS] Unknown tag warnings when building R docs with Roxygen 6.0.1

[SPARK-22430][R][DOCS] Unknown tag warnings when building R docs with Roxygen 6.0.1 ## What changes were proposed in this pull request? Removed export tag to get rid of unknown tag warnings ## How was this patch tested? Existing tests Author: Rekha Joshi Author: rjoshi2

spark-website git commit: update committer

Repository: spark-website Updated Branches: refs/heads/asf-site 32ff6fa97 -> 8bd24fb6d update committer Author: Felix Cheung Closes #103 from felixcheung/fc. Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit:

svn commit: r25438 - in /dev/spark/2.4.0-SNAPSHOT-2018_03_05_08_01-947b4e6-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Mon Mar 5 16:15:36 2018 New Revision: 25438 Log: Apache Spark 2.4.0-SNAPSHOT-2018_03_05_08_01-947b4e6 docs [This commit notification would consist of 1444 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23510][DOC][FOLLOW-UP] Update spark.sql.hive.metastore.version

Repository: spark Updated Branches: refs/heads/master a366b950b -> 947b4e6f0 [SPARK-23510][DOC][FOLLOW-UP] Update spark.sql.hive.metastore.version ## What changes were proposed in this pull request? Update `spark.sql.hive.metastore.version` to 2.3.2, same as HiveUtils.scala:

spark git commit: [SPARK-23329][SQL] Fix documentation of trigonometric functions

Repository: spark Updated Branches: refs/heads/branch-2.3 88dd335f6 -> 232b9f81f [SPARK-23329][SQL] Fix documentation of trigonometric functions ## What changes were proposed in this pull request? Provide more details in trigonometric function documentations. Referenced `java.lang.Math` for

spark git commit: [SPARK-23329][SQL] Fix documentation of trigonometric functions

Repository: spark Updated Branches: refs/heads/master 5ff72ffcf -> a366b950b [SPARK-23329][SQL] Fix documentation of trigonometric functions ## What changes were proposed in this pull request? Provide more details in trigonometric function documentations. Referenced `java.lang.Math` for

spark git commit: [SPARK-23566][MINOR][DOC] Argument name mismatch fixed

Repository: spark Updated Branches: refs/heads/master 42cf48e20 -> 5ff72ffcf [SPARK-23566][MINOR][DOC] Argument name mismatch fixed Argument name mismatch fixed. ## What changes were proposed in this pull request? `col` changed to `new` in doc string to match the argument list. Patch file

svn commit: r25432 - in /dev/spark/2.4.0-SNAPSHOT-2018_03_05_05_24-2ce37b5-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Mon Mar 5 13:37:51 2018 New Revision: 25432 Log: Apache Spark 2.4.0-SNAPSHOT-2018_03_05_05_24-2ce37b5 docs [This commit notification would consist of 1444 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23496][CORE] Locality of coalesced partitions can be severely skewed by the order of input partitions

Repository: spark Updated Branches: refs/heads/master 2ce37b50f -> 42cf48e20 [SPARK-23496][CORE] Locality of coalesced partitions can be severely skewed by the order of input partitions ## What changes were proposed in this pull request? The algorithm in

[1/2] spark git commit: [SPARK-23546][SQL] Refactor stateless methods/values in CodegenContext

Repository: spark Updated Branches: refs/heads/master 269cd5359 -> 2ce37b50f http://git-wip-us.apache.org/repos/asf/spark/blob/2ce37b50/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala

[2/2] spark git commit: [SPARK-23546][SQL] Refactor stateless methods/values in CodegenContext

[SPARK-23546][SQL] Refactor stateless methods/values in CodegenContext ## What changes were proposed in this pull request? A current `CodegenContext` class has immutable value or method without mutable state, too. This refactoring moves them to `CodeGenerator` object class which can be

svn commit: r25405 - in /dev/spark/2.4.0-SNAPSHOT-2018_03_05_00_01-269cd53-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Mon Mar 5 08:16:15 2018 New Revision: 25405 Log: Apache Spark 2.4.0-SNAPSHOT-2018_03_05_00_01-269cd53 docs [This commit notification would consist of 1444 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]