spark git commit: [SPARK-23052][SS] Migrate ConsoleSink to data source V2 api.

Repository: spark Updated Branches: refs/heads/branch-2.3 3a80cc59b -> 2a87c3a77 [SPARK-23052][SS] Migrate ConsoleSink to data source V2 api. ## What changes were proposed in this pull request? Migrate ConsoleSink to data source V2 api. Note that this includes a missing piece in

spark git commit: [SPARK-23052][SS] Migrate ConsoleSink to data source V2 api.

Repository: spark Updated Branches: refs/heads/master 39d244d92 -> 1c76a91e5 [SPARK-23052][SS] Migrate ConsoleSink to data source V2 api. ## What changes were proposed in this pull request? Migrate ConsoleSink to data source V2 api. Note that this includes a missing piece in

svn commit: r24281 - in /dev/spark/2.3.1-SNAPSHOT-2018_01_17_22_01-3a80cc5-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Jan 18 06:14:50 2018 New Revision: 24281 Log: Apache Spark 2.3.1-SNAPSHOT-2018_01_17_22_01-3a80cc5 docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23122][PYTHON][SQL] Deprecate register* for UDFs in SQLContext and Catalog in PySpark

Repository: spark Updated Branches: refs/heads/branch-2.3 f2688ef0f -> 3a80cc59b [SPARK-23122][PYTHON][SQL] Deprecate register* for UDFs in SQLContext and Catalog in PySpark ## What changes were proposed in this pull request? This PR proposes to deprecate `register*` for UDFs in

spark git commit: [SPARK-23122][PYTHON][SQL] Deprecate register* for UDFs in SQLContext and Catalog in PySpark

Repository: spark Updated Branches: refs/heads/master 021947020 -> 39d244d92 [SPARK-23122][PYTHON][SQL] Deprecate register* for UDFs in SQLContext and Catalog in PySpark ## What changes were proposed in this pull request? This PR proposes to deprecate `register*` for UDFs in `SQLContext`

svn commit: r24280 - in /dev/spark/2.4.0-SNAPSHOT-2018_01_17_20_01-0219470-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Jan 18 04:14:41 2018 New Revision: 24280 Log: Apache Spark 2.4.0-SNAPSHOT-2018_01_17_20_01-0219470 docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r24279 - in /dev/spark/2.3.1-SNAPSHOT-2018_01_17_18_01-f2688ef-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Jan 18 02:14:44 2018 New Revision: 24279 Log: Apache Spark 2.3.1-SNAPSHOT-2018_01_17_18_01-f2688ef docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-21996][SQL] read files with space in name for streaming

Repository: spark Updated Branches: refs/heads/master 1002bd6b2 -> 021947020 [SPARK-21996][SQL] read files with space in name for streaming ## What changes were proposed in this pull request? Structured streaming is now able to read files with space in file name (previously it would skip

spark git commit: [SPARK-23064][DOCS][SS] Added documentation for stream-stream joins

Repository: spark Updated Branches: refs/heads/branch-2.3 9783aea2c -> 050c1e24e [SPARK-23064][DOCS][SS] Added documentation for stream-stream joins ## What changes were proposed in this pull request? Added documentation for stream-stream joins

spark git commit: [SPARK-23064][DOCS][SS] Added documentation for stream-stream joins

Repository: spark Updated Branches: refs/heads/master bac0d661a -> 1002bd6b2 [SPARK-23064][DOCS][SS] Added documentation for stream-stream joins ## What changes were proposed in this pull request? Added documentation for stream-stream joins

spark git commit: [SPARK-23119][SS] Minor fixes to V2 streaming APIs

Repository: spark Updated Branches: refs/heads/master 7823d43ec -> bac0d661a [SPARK-23119][SS] Minor fixes to V2 streaming APIs ## What changes were proposed in this pull request? - Added `InterfaceStability.Evolving` annotations - Improved docs. ## How was this patch tested? Existing

spark git commit: [SPARK-23119][SS] Minor fixes to V2 streaming APIs

Repository: spark Updated Branches: refs/heads/branch-2.3 b84c2a306 -> 9783aea2c [SPARK-23119][SS] Minor fixes to V2 streaming APIs ## What changes were proposed in this pull request? - Added `InterfaceStability.Evolving` annotations - Improved docs. ## How was this patch tested? Existing

svn commit: r24278 - in /dev/spark/2.4.0-SNAPSHOT-2018_01_17_16_01-7823d43-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Thu Jan 18 00:14:46 2018 New Revision: 24278 Log: Apache Spark 2.4.0-SNAPSHOT-2018_01_17_16_01-7823d43 docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [MINOR] Fix typos in ML scaladocs

Repository: spark Updated Branches: refs/heads/master 45ad97df8 -> 7823d43ec [MINOR] Fix typos in ML scaladocs ## What changes were proposed in this pull request? Fixed some typos found in ML scaladocs ## How was this patch tested? NA Author: Bryan Cutler Closes

spark git commit: [SPARK-23132][PYTHON][ML] Run doctests in ml.image when testing

Repository: spark Updated Branches: refs/heads/branch-2.3 6e509fde3 -> b84c2a306 [SPARK-23132][PYTHON][ML] Run doctests in ml.image when testing ## What changes were proposed in this pull request? This PR proposes to actually run the doctests in `ml/image.py`. ## How was this patch tested?

spark git commit: [SPARK-23132][PYTHON][ML] Run doctests in ml.image when testing

Repository: spark Updated Branches: refs/heads/master 4e6f8fb15 -> 45ad97df8 [SPARK-23132][PYTHON][ML] Run doctests in ml.image when testing ## What changes were proposed in this pull request? This PR proposes to actually run the doctests in `ml/image.py`. ## How was this patch tested?

spark git commit: [SPARK-23047][PYTHON][SQL] Change MapVector to NullableMapVector in ArrowColumnVector

Repository: spark Updated Branches: refs/heads/branch-2.3 79ccd0cad -> 6e509fde3 [SPARK-23047][PYTHON][SQL] Change MapVector to NullableMapVector in ArrowColumnVector ## What changes were proposed in this pull request? This PR changes usage of `MapVector` in Spark codebase to use

spark git commit: [SPARK-23047][PYTHON][SQL] Change MapVector to NullableMapVector in ArrowColumnVector

Repository: spark Updated Branches: refs/heads/master e946c63dd -> 4e6f8fb15 [SPARK-23047][PYTHON][SQL] Change MapVector to NullableMapVector in ArrowColumnVector ## What changes were proposed in this pull request? This PR changes usage of `MapVector` in Spark codebase to use

svn commit: r24277 - in /dev/spark/2.3.1-SNAPSHOT-2018_01_17_14_01-79ccd0c-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Wed Jan 17 22:14:40 2018 New Revision: 24277 Log: Apache Spark 2.3.1-SNAPSHOT-2018_01_17_14_01-79ccd0c docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23093][SS] Don't change run id when reconfiguring a continuous processing query.

Repository: spark Updated Branches: refs/heads/branch-2.3 dbd2a5566 -> 79ccd0cad [SPARK-23093][SS] Don't change run id when reconfiguring a continuous processing query. ## What changes were proposed in this pull request? Keep the run ID static, using a different ID for the epoch coordinator

spark git commit: [SPARK-23093][SS] Don't change run id when reconfiguring a continuous processing query.

Repository: spark Updated Branches: refs/heads/master 86a845031 -> e946c63dd [SPARK-23093][SS] Don't change run id when reconfiguring a continuous processing query. ## What changes were proposed in this pull request? Keep the run ID static, using a different ID for the epoch coordinator to

spark git commit: [SPARK-23033][SS] Don't use task level retry for continuous processing

Repository: spark Updated Branches: refs/heads/branch-2.3 1a6dfaf25 -> dbd2a5566 [SPARK-23033][SS] Don't use task level retry for continuous processing ## What changes were proposed in this pull request? Continuous processing tasks will fail on any attempt number greater than 0.

spark git commit: [SPARK-23033][SS] Don't use task level retry for continuous processing

Repository: spark Updated Branches: refs/heads/master c132538a1 -> 86a845031 [SPARK-23033][SS] Don't use task level retry for continuous processing ## What changes were proposed in this pull request? Continuous processing tasks will fail on any attempt number greater than 0.

svn commit: r24272 - in /dev/spark/2.4.0-SNAPSHOT-2018_01_17_12_01-c132538-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Wed Jan 17 20:14:42 2018 New Revision: 24272 Log: Apache Spark 2.4.0-SNAPSHOT-2018_01_17_12_01-c132538 docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r24269 - in /dev/spark/2.3.1-SNAPSHOT-2018_01_17_10_01-1a6dfaf-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Wed Jan 17 18:15:29 2018 New Revision: 24269 Log: Apache Spark 2.3.1-SNAPSHOT-2018_01_17_10_01-1a6dfaf docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23020] Ignore Flaky Test: SparkLauncherSuite.testInProcessLauncher

Repository: spark Updated Branches: refs/heads/branch-2.3 aae73a21a -> 1a6dfaf25 [SPARK-23020] Ignore Flaky Test: SparkLauncherSuite.testInProcessLauncher ## What changes were proposed in this pull request? Temporarily ignoring flaky test `SparkLauncherSuite.testInProcessLauncher` to

spark git commit: [SPARK-23020] Ignore Flaky Test: SparkLauncherSuite.testInProcessLauncher

Repository: spark Updated Branches: refs/heads/master 8598a982b -> c132538a1 [SPARK-23020] Ignore Flaky Test: SparkLauncherSuite.testInProcessLauncher ## What changes were proposed in this pull request? Temporarily ignoring flaky test `SparkLauncherSuite.testInProcessLauncher` to de-flake

svn commit: r24260 - in /dev/spark/2.4.0-SNAPSHOT-2018_01_17_08_01-0f8a286-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Wed Jan 17 16:17:13 2018 New Revision: 24260 Log: Apache Spark 2.4.0-SNAPSHOT-2018_01_17_08_01-0f8a286 docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23079][SQL] Fix query constraints propagation with aliases

Repository: spark Updated Branches: refs/heads/branch-2.3 cbb6bda43 -> aae73a21a [SPARK-23079][SQL] Fix query constraints propagation with aliases ## What changes were proposed in this pull request? Previously, PR #19201 fix the problem of non-converging constraints. After that PR #19149

svn commit: r24258 - in /dev/spark/2.3.1-SNAPSHOT-2018_01_17_06_01-cbb6bda-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Wed Jan 17 14:14:54 2018 New Revision: 24258 Log: Apache Spark 2.3.1-SNAPSHOT-2018_01_17_06_01-cbb6bda docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-21783][SQL] Turn on ORC filter push-down by default

Repository: spark Updated Branches: refs/heads/branch-2.3 bfbc2d41b -> cbb6bda43 [SPARK-21783][SQL] Turn on ORC filter push-down by default ## What changes were proposed in this pull request? ORC filter push-down is disabled by default from the beginning,

spark git commit: [SPARK-21783][SQL] Turn on ORC filter push-down by default

Repository: spark Updated Branches: refs/heads/master 1f3d933e0 -> 0f8a28617 [SPARK-21783][SQL] Turn on ORC filter push-down by default ## What changes were proposed in this pull request? ORC filter push-down is disabled by default from the beginning,

svn commit: r24255 - in /dev/spark/2.4.0-SNAPSHOT-2018_01_17_04_01-1f3d933-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Wed Jan 17 12:19:25 2018 New Revision: 24255 Log: Apache Spark 2.4.0-SNAPSHOT-2018_01_17_04_01-1f3d933 docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r24254 - in /dev/spark/2.3.1-SNAPSHOT-2018_01_17_02_01-bfbc2d4-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Wed Jan 17 10:15:04 2018 New Revision: 24254 Log: Apache Spark 2.3.1-SNAPSHOT-2018_01_17_02_01-bfbc2d4 docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r24253 - in /dev/spark/2.4.0-SNAPSHOT-2018_01_17_00_01-a0aedb0-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Wed Jan 17 08:15:25 2018 New Revision: 24253 Log: Apache Spark 2.4.0-SNAPSHOT-2018_01_17_00_01-a0aedb0 docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23062][SQL] Improve EXCEPT documentation

Repository: spark Updated Branches: refs/heads/branch-2.3 8ef323c57 -> bfbc2d41b [SPARK-23062][SQL] Improve EXCEPT documentation ## What changes were proposed in this pull request? Make the default behavior of EXCEPT (i.e. EXCEPT DISTINCT) more explicit in the documentation, and call out the

spark git commit: [SPARK-23062][SQL] Improve EXCEPT documentation

Repository: spark Updated Branches: refs/heads/master a0aedb0de -> 1f3d933e0 [SPARK-23062][SQL] Improve EXCEPT documentation ## What changes were proposed in this pull request? Make the default behavior of EXCEPT (i.e. EXCEPT DISTINCT) more explicit in the documentation, and call out the