spark git commit: [SPARK-22274][PYTHON][SQL] User-defined aggregation functions with pandas udf (full shuffle)

Repository: spark Updated Branches: refs/heads/master 51eb75026 -> b2ce17b4c [SPARK-22274][PYTHON][SQL] User-defined aggregation functions with pandas udf (full shuffle) ## What changes were proposed in this pull request? Add support for using pandas UDFs with groupby().agg(). This PR

svn commit: r24370 - in /dev/spark/2.3.1-SNAPSHOT-2018_01_22_18_01-7241556-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Tue Jan 23 02:15:25 2018 New Revision: 24370 Log: Apache Spark 2.3.1-SNAPSHOT-2018_01_22_18_01-7241556 docs [This commit notification would consist of 1442 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r24368 - in /dev/spark/2.4.0-SNAPSHOT-2018_01_22_16_01-51eb750-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Tue Jan 23 00:14:52 2018 New Revision: 24368 Log: Apache Spark 2.4.0-SNAPSHOT-2018_01_22_16_01-51eb750 docs [This commit notification would consist of 1442 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-22389][SQL] data source v2 partitioning reporting interface

Repository: spark Updated Branches: refs/heads/master 76b8b840d -> 51eb75026 [SPARK-22389][SQL] data source v2 partitioning reporting interface ## What changes were proposed in this pull request? a new interface which allows data source to report partitioning and avoid shuffle at Spark

spark git commit: [SPARK-22389][SQL] data source v2 partitioning reporting interface

Repository: spark Updated Branches: refs/heads/branch-2.3 566ef93a6 -> 7241556d8 [SPARK-22389][SQL] data source v2 partitioning reporting interface ## What changes were proposed in this pull request? a new interface which allows data source to report partitioning and avoid shuffle at Spark

svn commit: r24366 - in /dev/spark/2.3.1-SNAPSHOT-2018_01_22_14_01-566ef93-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Mon Jan 22 22:14:49 2018 New Revision: 24366 Log: Apache Spark 2.3.1-SNAPSHOT-2018_01_22_14_01-566ef93 docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r24365 - /dev/spark/KEYS

Author: sameerag Date: Mon Jan 22 21:25:48 2018 New Revision: 24365 Log: Update KEYS Modified: dev/spark/KEYS Modified: dev/spark/KEYS == --- dev/spark/KEYS (original) +++ dev/spark/KEYS Mon Jan 22 21:25:48 2018 @@

svn commit: r24364 - in /dev/spark/v2.3.0-rc2-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _site/api/java/org/apache/spark

Author: sameerag Date: Mon Jan 22 20:30:45 2018 New Revision: 24364 Log: Apache Spark v2.3.0-rc2 docs [This commit notification would consist of 1444 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r24363 - in /dev/spark/2.4.0-SNAPSHOT-2018_01_22_12_01-76b8b84-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Mon Jan 22 20:17:30 2018 New Revision: 24363 Log: Apache Spark 2.4.0-SNAPSHOT-2018_01_22_12_01-76b8b84 docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [MINOR] Typo fixes

Repository: spark Updated Branches: refs/heads/master 446948af1 -> 76b8b840d [MINOR] Typo fixes ## What changes were proposed in this pull request? Typo fixes ## How was this patch tested? Local build / Doc-only changes Author: Jacek Laskowski Closes #20344 from

spark git commit: [MINOR] Typo fixes

Repository: spark Updated Branches: refs/heads/branch-2.3 6facc7fb2 -> 566ef93a6 [MINOR] Typo fixes ## What changes were proposed in this pull request? Typo fixes ## How was this patch tested? Local build / Doc-only changes Author: Jacek Laskowski Closes #20344 from

svn commit: r24362 - /dev/spark/v2.3.0-rc2-bin/

Author: sameerag Date: Mon Jan 22 19:45:22 2018 New Revision: 24362 Log: Apache Spark v2.3.0-rc2 Added: dev/spark/v2.3.0-rc2-bin/ dev/spark/v2.3.0-rc2-bin/SparkR_2.3.0.tar.gz (with props) dev/spark/v2.3.0-rc2-bin/SparkR_2.3.0.tar.gz.asc

[1/2] spark git commit: Preparing Spark release v2.3.0-rc2

Repository: spark Updated Branches: refs/heads/branch-2.3 4e75b0cb4 -> 6facc7fb2 Preparing Spark release v2.3.0-rc2 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/489ecb0e Tree:

[spark] Git Push Summary

Repository: spark Updated Tags: refs/tags/v2.3.0-rc2 [created] 489ecb0ef - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org

[2/2] spark git commit: Preparing development version 2.3.1-SNAPSHOT

Preparing development version 2.3.1-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6facc7fb Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6facc7fb Diff:

spark git commit: [SPARK-23121][CORE] Fix for ui becoming unaccessible for long running streaming apps

Repository: spark Updated Branches: refs/heads/master 4327ccf28 -> 446948af1 [SPARK-23121][CORE] Fix for ui becoming unaccessible for long running streaming apps ## What changes were proposed in this pull request? The allJobs and the job pages attempt to use stage attempt and DAG

spark git commit: [SPARK-23121][CORE] Fix for ui becoming unaccessible for long running streaming apps

Repository: spark Updated Branches: refs/heads/branch-2.3 d963ba031 -> 4e75b0cb4 [SPARK-23121][CORE] Fix for ui becoming unaccessible for long running streaming apps ## What changes were proposed in this pull request? The allJobs and the job pages attempt to use stage attempt and DAG

svn commit: r24360 - in /dev/spark/2.4.0-SNAPSHOT-2018_01_22_08_01-4327ccf-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Mon Jan 22 16:22:00 2018 New Revision: 24360 Log: Apache Spark 2.4.0-SNAPSHOT-2018_01_22_08_01-4327ccf docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-11630][CORE] ClosureCleaner moved from warning to debug

Repository: spark Updated Branches: refs/heads/master 87ffe7add -> 4327ccf28 [SPARK-11630][CORE] ClosureCleaner moved from warning to debug ## What changes were proposed in this pull request? ClosureCleaner moved from warning to debug ## How was this patch tested? Existing tests Author:

svn commit: r24354 - in /dev/spark/2.3.1-SNAPSHOT-2018_01_22_06_01-d963ba0-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Mon Jan 22 14:20:14 2018 New Revision: 24354 Log: Apache Spark 2.3.1-SNAPSHOT-2018_01_22_06_01-d963ba0 docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-7721][PYTHON][TESTS] Adds PySpark coverage generation script

Repository: spark Updated Branches: refs/heads/master 5d680cae4 -> 87ffe7add [SPARK-7721][PYTHON][TESTS] Adds PySpark coverage generation script ## What changes were proposed in this pull request? Note that this PR was made based on the top of https://github.com/apache/spark/pull/20151. So,

spark git commit: [SPARK-23090][SQL] polish ColumnVector

Repository: spark Updated Branches: refs/heads/branch-2.3 1069fad41 -> d963ba031 [SPARK-23090][SQL] polish ColumnVector ## What changes were proposed in this pull request? Several improvements: * provide a default implementation for the batch get methods * rename `getChildColumn` to

spark git commit: [SPARK-23090][SQL] polish ColumnVector

Repository: spark Updated Branches: refs/heads/master 896e45af5 -> 5d680cae4 [SPARK-23090][SQL] polish ColumnVector ## What changes were proposed in this pull request? Several improvements: * provide a default implementation for the batch get methods * rename `getChildColumn` to `getChild`,

spark git commit: [MINOR][SQL][TEST] Test case cleanups for recent PRs

Repository: spark Updated Branches: refs/heads/master 78801881c -> 896e45af5 [MINOR][SQL][TEST] Test case cleanups for recent PRs ## What changes were proposed in this pull request? Revert the unneeded test case changes we made in SPARK-23000 Also fixes the test suites that do not call

spark git commit: [MINOR][SQL][TEST] Test case cleanups for recent PRs

Repository: spark Updated Branches: refs/heads/branch-2.3 d933fcea6 -> 1069fad41 [MINOR][SQL][TEST] Test case cleanups for recent PRs ## What changes were proposed in this pull request? Revert the unneeded test case changes we made in SPARK-23000 Also fixes the test suites that do not call

spark git commit: [SPARK-23170][SQL] Dump the statistics of effective runs of analyzer and optimizer rules

Repository: spark Updated Branches: refs/heads/master 73281161f -> 78801881c [SPARK-23170][SQL] Dump the statistics of effective runs of analyzer and optimizer rules ## What changes were proposed in this pull request? Dump the statistics of effective runs of analyzer and optimizer rules.

spark git commit: [SPARK-23170][SQL] Dump the statistics of effective runs of analyzer and optimizer rules

Repository: spark Updated Branches: refs/heads/branch-2.3 743b9173f -> d933fcea6 [SPARK-23170][SQL] Dump the statistics of effective runs of analyzer and optimizer rules ## What changes were proposed in this pull request? Dump the statistics of effective runs of analyzer and optimizer

spark git commit: [SPARK-23122][PYSPARK][FOLLOW-UP] Update the docs for UDF Registration

Repository: spark Updated Branches: refs/heads/branch-2.3 cf078a205 -> 743b9173f [SPARK-23122][PYSPARK][FOLLOW-UP] Update the docs for UDF Registration ## What changes were proposed in this pull request? This PR is to update the docs for UDF registration ## How was this patch tested? N/A

spark git commit: [SPARK-23122][PYSPARK][FOLLOW-UP] Update the docs for UDF Registration

Repository: spark Updated Branches: refs/heads/master 60175e959 -> 73281161f [SPARK-23122][PYSPARK][FOLLOW-UP] Update the docs for UDF Registration ## What changes were proposed in this pull request? This PR is to update the docs for UDF registration ## How was this patch tested? N/A

spark git commit: [MINOR][DOC] Fix the path to the examples jar

Repository: spark Updated Branches: refs/heads/branch-2.3 57c320a0d -> cf078a205 [MINOR][DOC] Fix the path to the examples jar ## What changes were proposed in this pull request? The example jar file is now in ./examples/jars directory of Spark distribution. Author: Arseniy Tashoyan

spark git commit: [MINOR][DOC] Fix the path to the examples jar

Repository: spark Updated Branches: refs/heads/master ec2289761 -> 60175e959 [MINOR][DOC] Fix the path to the examples jar ## What changes were proposed in this pull request? The example jar file is now in ./examples/jars directory of Spark distribution. Author: Arseniy Tashoyan

svn commit: r24352 - in /dev/spark/2.3.1-SNAPSHOT-2018_01_22_02_01-57c320a-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Mon Jan 22 10:15:16 2018 New Revision: 24352 Log: Apache Spark 2.3.1-SNAPSHOT-2018_01_22_02_01-57c320a docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r24350 - in /dev/spark/2.4.0-SNAPSHOT-2018_01_22_00_01-ec22897-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

Author: pwendell Date: Mon Jan 22 08:16:21 2018 New Revision: 24350 Log: Apache Spark 2.4.0-SNAPSHOT-2018_01_22_00_01-ec22897 docs [This commit notification would consist of 1441 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]