spark git commit: [SPARK-22274][PYTHON][SQL][FOLLOWUP] Use `assertRaisesRegexp` instead of `assertRaisesRegex`.

Repository: spark Updated Branches: refs/heads/master 4b7cd479a -> 07cee3373 [SPARK-22274][PYTHON][SQL][FOLLOWUP] Use `assertRaisesRegexp` instead of `assertRaisesRegex`. ## What changes were proposed in this pull request? This is a follow-up pr of #19872 which uses `assertRaisesRegex` but

svn commit: r24614 - in /dev/spark/2.3.1-SNAPSHOT-2018_01_31_22_01-871fd48-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Feb 1 06:16:10 2018 New Revision: 24614 Log: Apache Spark 2.3.1-SNAPSHOT-2018_01_31_22_01-871fd48 docs [This commit notification would consist of 1442 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: Revert "[SPARK-23200] Reset Kubernetes-specific config on Checkpoint restore"

Repository: spark Updated Branches: refs/heads/master b6b50efc8 -> 4b7cd479a Revert "[SPARK-23200] Reset Kubernetes-specific config on Checkpoint restore" This reverts commit d1721816d26bedee3c72eeb75db49da500568376. The patch is not fully tested and out-of-date. So revert it. Project:

spark git commit: [SQL][MINOR] Inline SpecifiedWindowFrame.defaultWindowFrame().

Repository: spark Updated Branches: refs/heads/master cc41245fa -> b6b50efc8 [SQL][MINOR] Inline SpecifiedWindowFrame.defaultWindowFrame(). ## What changes were proposed in this pull request? SpecifiedWindowFrame.defaultWindowFrame(hasOrderSpecification, acceptWindowFrame) was designed to

spark git commit: [SPARK-23188][SQL] Make vectorized columar reader batch size configurable

Repository: spark Updated Branches: refs/heads/master b2e7677f4 -> cc41245fa [SPARK-23188][SQL] Make vectorized columar reader batch size configurable ## What changes were proposed in this pull request? This PR include the following changes: - Make the capacity of

spark git commit: [SPARK-21396][SQL] Fixes MatchError when UDTs are passed through Hive Thriftserver

Repository: spark Updated Branches: refs/heads/master 56ae32657 -> b2e7677f4 [SPARK-21396][SQL] Fixes MatchError when UDTs are passed through Hive Thriftserver Signed-off-by: Atallah Hezbor ## What changes were proposed in this pull request? This PR proposes modifying the match statement

spark git commit: [SPARK-21396][SQL] Fixes MatchError when UDTs are passed through Hive Thriftserver

Repository: spark Updated Branches: refs/heads/branch-2.3 59e89a299 -> 871fd48dc [SPARK-21396][SQL] Fixes MatchError when UDTs are passed through Hive Thriftserver Signed-off-by: Atallah Hezbor ## What changes were proposed in this pull request? This PR proposes modifying the match

[1/2] spark git commit: [SPARK-23268][SQL] Reorganize packages in data source V2

Repository: spark Updated Branches: refs/heads/master 2ac895be9 -> 56ae32657 http://git-wip-us.apache.org/repos/asf/spark/blob/56ae3265/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/RateSourceProvider.scala

[2/2] spark git commit: [SPARK-23268][SQL] Reorganize packages in data source V2

[SPARK-23268][SQL] Reorganize packages in data source V2 ## What changes were proposed in this pull request? 1. create a new package for partitioning/distribution related classes. As Spark will add new concrete implementations of `Distribution` in new releases, it is good to have a new

[1/2] spark git commit: [SPARK-23268][SQL] Reorganize packages in data source V2

Repository: spark Updated Branches: refs/heads/branch-2.3 0d0f57936 -> 59e89a299 http://git-wip-us.apache.org/repos/asf/spark/blob/59e89a29/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/RateSourceProvider.scala

[2/2] spark git commit: [SPARK-23268][SQL] Reorganize packages in data source V2

[SPARK-23268][SQL] Reorganize packages in data source V2 ## What changes were proposed in this pull request? 1. create a new package for partitioning/distribution related classes. As Spark will add new concrete implementations of `Distribution` in new releases, it is good to have a new

svn commit: r24611 - in /dev/spark/2.4.0-SNAPSHOT-2018_01_31_20_01-52e00f7-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Feb 1 04:15:49 2018 New Revision: 24611 Log: Apache Spark 2.4.0-SNAPSHOT-2018_01_31_20_01-52e00f7 docs [This commit notification would consist of 1444 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23247][SQL] combines Unsafe operations and statistics operations in Scan Data Source

Repository: spark Updated Branches: refs/heads/master 52e00f706 -> 2ac895be9 [SPARK-23247][SQL] combines Unsafe operations and statistics operations in Scan Data Source ## What changes were proposed in this pull request? Currently, we scan the execution plan of the data source, first the

spark git commit: [SPARK-23280][SQL] add map type support to ColumnVector

Repository: spark Updated Branches: refs/heads/branch-2.3 7ccfc7530 -> 0d0f57936 [SPARK-23280][SQL] add map type support to ColumnVector ## What changes were proposed in this pull request? Fill the last missing piece of `ColumnVector`: the map type support. The idea is similar to the array

spark git commit: [SPARK-23280][SQL] add map type support to ColumnVector

Repository: spark Updated Branches: refs/heads/master f470df2fc -> 52e00f706 [SPARK-23280][SQL] add map type support to ColumnVector ## What changes were proposed in this pull request? Fill the last missing piece of `ColumnVector`: the map type support. The idea is similar to the array type

spark git commit: [SPARK-23157][SQL][FOLLOW-UP] DataFrame -> SparkDataFrame in R comment

Repository: spark Updated Branches: refs/heads/branch-2.3 8ee3a71c9 -> 7ccfc7530 [SPARK-23157][SQL][FOLLOW-UP] DataFrame -> SparkDataFrame in R comment Author: Henry Robinson Closes #20443 from henryr/SPARK-23157. (cherry picked from commit

spark git commit: [SPARK-23157][SQL][FOLLOW-UP] DataFrame -> SparkDataFrame in R comment

Repository: spark Updated Branches: refs/heads/master 9ff1d96f0 -> f470df2fc [SPARK-23157][SQL][FOLLOW-UP] DataFrame -> SparkDataFrame in R comment Author: Henry Robinson Closes #20443 from henryr/SPARK-23157. Project: http://git-wip-us.apache.org/repos/asf/spark/repo

svn commit: r24607 - in /dev/spark/2.4.0-SNAPSHOT-2018_01_31_16_01-9ff1d96-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Feb 1 00:16:03 2018 New Revision: 24607 Log: Apache Spark 2.4.0-SNAPSHOT-2018_01_31_16_01-9ff1d96 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r24606 - in /dev/spark/2.3.1-SNAPSHOT-2018_01_31_14_01-8ee3a71-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Wed Jan 31 22:15:35 2018 New Revision: 24606 Log: Apache Spark 2.3.1-SNAPSHOT-2018_01_31_14_01-8ee3a71 docs [This commit notification would consist of 1442 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: revert the removal of import in SPARK-23281

Repository: spark Updated Branches: refs/heads/branch-2.2 5273cc791 -> cb73ecd2f revert the removal of import in SPARK-23281 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/cb73ecd2 Tree:

spark git commit: [SPARK-23281][SQL] Query produces results in incorrect order when a composite order by clause refers to both original columns and aliases

Repository: spark Updated Branches: refs/heads/branch-2.2 0e58fee9d -> 5273cc791 [SPARK-23281][SQL] Query produces results in incorrect order when a composite order by clause refers to both original columns and aliases ## What changes were proposed in this pull request? Here is the test

spark git commit: [SPARK-23281][SQL] Query produces results in incorrect order when a composite order by clause refers to both original columns and aliases

Repository: spark Updated Branches: refs/heads/branch-2.3 f5f21e8c4 -> 8ee3a71c9 [SPARK-23281][SQL] Query produces results in incorrect order when a composite order by clause refers to both original columns and aliases ## What changes were proposed in this pull request? Here is the test

spark git commit: [SPARK-23281][SQL] Query produces results in incorrect order when a composite order by clause refers to both original columns and aliases

Repository: spark Updated Branches: refs/heads/master dd242bad3 -> 9ff1d96f0 [SPARK-23281][SQL] Query produces results in incorrect order when a composite order by clause refers to both original columns and aliases ## What changes were proposed in this pull request? Here is the test snippet.

svn commit: r24605 - in /dev/spark/2.4.0-SNAPSHOT-2018_01_31_12_01-dd242ba-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Wed Jan 31 20:16:23 2018 New Revision: 24605 Log: Apache Spark 2.4.0-SNAPSHOT-2018_01_31_12_01-dd242ba docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-21525][STREAMING] Check error code from supervisor RPC.

Repository: spark Updated Branches: refs/heads/master 8c21170de -> dd242bad3 [SPARK-21525][STREAMING] Check error code from supervisor RPC. The code was ignoring the error code from the AddBlock RPC, which means that a failure to write to the WAL was being ignored by the receiver, and would

spark git commit: [SPARK-23249][SQL] Improved block merging logic for partitions

Repository: spark Updated Branches: refs/heads/branch-2.3 33f17b28b -> f5f21e8c4 [SPARK-23249][SQL] Improved block merging logic for partitions ## What changes were proposed in this pull request? Change DataSourceScanExec so that when grouping blocks together into partitions, also checks

spark git commit: [SPARK-23249][SQL] Improved block merging logic for partitions

Repository: spark Updated Branches: refs/heads/master 48dd6a4c7 -> 8c21170de [SPARK-23249][SQL] Improved block merging logic for partitions ## What changes were proposed in this pull request? Change DataSourceScanExec so that when grouping blocks together into partitions, also checks the

spark git commit: revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSet

Repository: spark Updated Branches: refs/heads/branch-2.3 c83246c9a -> 33f17b28b revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSet ## What changes were proposed in this pull request? In https://github.com/apache/spark/pull/19980 , we thought `anyNullsSet` can be simply implemented

spark git commit: revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSet

Repository: spark Updated Branches: refs/heads/master 3d0911bbe -> 48dd6a4c7 revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSet ## What changes were proposed in this pull request? In https://github.com/apache/spark/pull/19980 , we thought `anyNullsSet` can be simply implemented by

svn commit: r24599 - in /dev/spark/2.4.0-SNAPSHOT-2018_01_31_04_01-3d0911b-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Wed Jan 31 12:20:49 2018 New Revision: 24599 Log: Apache Spark 2.4.0-SNAPSHOT-2018_01_31_04_01-3d0911b docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23228][PYSPARK] Add Python Created jsparkSession to JVM's defaultSession

Repository: spark Updated Branches: refs/heads/master 161a3f2ae -> 3d0911bbe [SPARK-23228][PYSPARK] Add Python Created jsparkSession to JVM's defaultSession ## What changes were proposed in this pull request? In the current PySpark code, Python created `jsparkSession` doesn't add to JVM's

svn commit: r24596 - in /dev/spark/2.3.1-SNAPSHOT-2018_01_31_02_01-c83246c-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Wed Jan 31 10:16:04 2018 New Revision: 24596 Log: Apache Spark 2.3.1-SNAPSHOT-2018_01_31_02_01-c83246c docs [This commit notification would consist of 1442 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23112][DOC] Update ML migration guide with breaking and behavior changes.

Repository: spark Updated Branches: refs/heads/branch-2.3 7ec8ad7ba -> c83246c9a [SPARK-23112][DOC] Update ML migration guide with breaking and behavior changes. Add breaking changes, as well as update behavior changes, to `2.3` ML migration guide. ## How was this patch tested? Doc only

spark git commit: [SPARK-23112][DOC] Update ML migration guide with breaking and behavior changes.

Repository: spark Updated Branches: refs/heads/master 695f7146b -> 161a3f2ae [SPARK-23112][DOC] Update ML migration guide with breaking and behavior changes. Add breaking changes, as well as update behavior changes, to `2.3` ML migration guide. ## How was this patch tested? Doc only

svn commit: r24592 - in /dev/spark/2.4.0-SNAPSHOT-2018_01_31_00_01-695f714-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Wed Jan 31 08:16:05 2018 New Revision: 24592 Log: Apache Spark 2.4.0-SNAPSHOT-2018_01_31_00_01-695f714 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]