svn commit: r30922 - in /dev/spark/3.0.0-SNAPSHOT-2018_11_15_21_50-4ac8f9b-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-15 Thread pwendell
Author: pwendell Date: Fri Nov 16 06:02:40 2018 New Revision: 30922 Log: Apache Spark 3.0.0-SNAPSHOT-2018_11_15_21_50-4ac8f9b docs [This commit notification would consist of 1755 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.enableRadixSort` instead of `SparkEnv.get.conf.get(SQLConf.RADIX_SORT_ENABLED)`.

2018-11-15 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.4 6148a77a5 -> 2d67be932 [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.enableRadixSort` instead of `SparkEnv.get.conf.get(SQLConf.RADIX_SORT_ENABLED)`. ## What changes were proposed in this pull request? This is a follow-up of #20393.

spark git commit: [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.enableRadixSort` instead of `SparkEnv.get.conf.get(SQLConf.RADIX_SORT_ENABLED)`.

2018-11-15 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 3649fe599 -> dad2d826a [SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.enableRadixSort` instead of `SparkEnv.get.conf.get(SQLConf.RADIX_SORT_ENABLED)`. ## What changes were proposed in this pull request? This is a follow-up of #20393. We

svn commit: r30921 - in /dev/spark/2.4.1-SNAPSHOT-2018_11_15_19_48-6148a77-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-15 Thread pwendell
Author: pwendell Date: Fri Nov 16 04:03:13 2018 New Revision: 30921 Log: Apache Spark 2.4.1-SNAPSHOT-2018_11_15_19_48-6148a77 docs [This commit notification would consist of 1476 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r30919 - in /dev/spark/3.0.0-SNAPSHOT-2018_11_15_17_46-3649fe5-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-15 Thread pwendell
Author: pwendell Date: Fri Nov 16 01:58:54 2018 New Revision: 30919 Log: Apache Spark 3.0.0-SNAPSHOT-2018_11_15_17_46-3649fe5 docs [This commit notification would consist of 1755 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25883][BACKPORT][SQL][MINOR] Override method `prettyName` in `from_avro`/`to_avro`

2018-11-15 Thread gurwls223
Repository: spark Updated Branches: refs/heads/branch-2.4 96834fb77 -> 6148a77a5 [SPARK-25883][BACKPORT][SQL][MINOR] Override method `prettyName` in `from_avro`/`to_avro` Back port https://github.com/apache/spark/pull/22890 to branch-2.4. It is a bug fix for this issue:

[1/2] spark git commit: [SPARK-26035][PYTHON] Break large streaming/tests.py files into smaller files

2018-11-15 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 9a5fda60e -> 3649fe599 http://git-wip-us.apache.org/repos/asf/spark/blob/3649fe59/python/pyspark/streaming/tests/test_listener.py -- diff --git

[2/2] spark git commit: [SPARK-26035][PYTHON] Break large streaming/tests.py files into smaller files

2018-11-15 Thread gurwls223
[SPARK-26035][PYTHON] Break large streaming/tests.py files into smaller files ## What changes were proposed in this pull request? This PR continues to break down a big large file into smaller files. See https://github.com/apache/spark/pull/23021. It targets to follow

svn commit: r30918 - in /dev/spark/2.4.1-SNAPSHOT-2018_11_15_11_32-96834fb-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-15 Thread pwendell
Author: pwendell Date: Thu Nov 15 19:49:04 2018 New Revision: 30918 Log: Apache Spark 2.4.1-SNAPSHOT-2018_11_15_11_32-96834fb docs [This commit notification would consist of 1476 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r30917 - in /dev/spark/2.3.3-SNAPSHOT-2018_11_15_11_32-7a59618-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-15 Thread pwendell
Author: pwendell Date: Thu Nov 15 19:47:16 2018 New Revision: 30917 Log: Apache Spark 2.3.3-SNAPSHOT-2018_11_15_11_32-7a59618 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r30916 - in /dev/spark/3.0.0-SNAPSHOT-2018_11_15_09_22-9a5fda6-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-15 Thread pwendell
Author: pwendell Date: Thu Nov 15 17:34:48 2018 New Revision: 30916 Log: Apache Spark 3.0.0-SNAPSHOT-2018_11_15_09_22-9a5fda6 docs [This commit notification would consist of 1755 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-26011][SPARK-SUBMIT] Yarn mode pyspark app without python main resource does not honor "spark.jars.packages"

2018-11-15 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.3 0c7d82b1c -> 7a596187e [SPARK-26011][SPARK-SUBMIT] Yarn mode pyspark app without python main resource does not honor "spark.jars.packages" SparkSubmit determines pyspark app by the suffix of primary resource but Livy uses

spark git commit: [SPARK-26011][SPARK-SUBMIT] Yarn mode pyspark app without python main resource does not honor "spark.jars.packages"

2018-11-15 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.4 aaa21d812 -> 96834fb77 [SPARK-26011][SPARK-SUBMIT] Yarn mode pyspark app without python main resource does not honor "spark.jars.packages" SparkSubmit determines pyspark app by the suffix of primary resource but Livy uses

spark git commit: [SPARK-26011][SPARK-SUBMIT] Yarn mode pyspark app without python main resource does not honor "spark.jars.packages"

2018-11-15 Thread srowen
Repository: spark Updated Branches: refs/heads/master cae5879db -> 9a5fda60e [SPARK-26011][SPARK-SUBMIT] Yarn mode pyspark app without python main resource does not honor "spark.jars.packages" SparkSubmit determines pyspark app by the suffix of primary resource but Livy uses "spark-internal"

spark git commit: [SPARK-26044][WEBUI] Aggregated Metrics table sort based on executor ID

2018-11-15 Thread srowen
Repository: spark Updated Branches: refs/heads/master 91405b3b6 -> cae5879db [SPARK-26044][WEBUI] Aggregated Metrics table sort based on executor ID ## What changes were proposed in this pull request? Aggregated Metrics table in the stage page is not sorted based on the executorID properly.

svn commit: r30914 - in /dev/spark/2.4.1-SNAPSHOT-2018_11_15_07_08-aaa21d8-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-15 Thread pwendell
Author: pwendell Date: Thu Nov 15 15:23:05 2018 New Revision: 30914 Log: Apache Spark 2.4.1-SNAPSHOT-2018_11_15_07_08-aaa21d8 docs [This commit notification would consist of 1476 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-22450][WIP][CORE][MLLIB][FOLLOWUP] Safely register MultivariateGaussian

2018-11-15 Thread srowen
Repository: spark Updated Branches: refs/heads/master 9610efc25 -> 91405b3b6 [SPARK-22450][WIP][CORE][MLLIB][FOLLOWUP] Safely register MultivariateGaussian ## What changes were proposed in this pull request? register following classes in Kryo:

svn commit: r30913 - in /dev/spark/3.0.0-SNAPSHOT-2018_11_15_04_58-9610efc-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-15 Thread pwendell
Author: pwendell Date: Thu Nov 15 13:11:21 2018 New Revision: 30913 Log: Apache Spark 3.0.0-SNAPSHOT-2018_11_15_04_58-9610efc docs [This commit notification would consist of 1755 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-26055][CORE] InterfaceStability annotations should be retained at runtime

2018-11-15 Thread wenchen
Repository: spark Updated Branches: refs/heads/master b46f75a5a -> 9610efc25 [SPARK-26055][CORE] InterfaceStability annotations should be retained at runtime ## What changes were proposed in this pull request? It's good to have annotations available at runtime, so that tools like MiMa can

spark git commit: [SPARK-26057][SQL] Transform also analyzed plans when dedup references

2018-11-15 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.4 ba638a783 -> aaa21d812 [SPARK-26057][SQL] Transform also analyzed plans when dedup references ## What changes were proposed in this pull request? In SPARK-24865 `AnalysisBarrier` was removed and in order to improve resolution speed,

spark git commit: [SPARK-26057][SQL] Transform also analyzed plans when dedup references

2018-11-15 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 44d4ef60b -> b46f75a5a [SPARK-26057][SQL] Transform also analyzed plans when dedup references ## What changes were proposed in this pull request? In SPARK-24865 `AnalysisBarrier` was removed and in order to improve resolution speed, the

spark git commit: [SPARK-25974][SQL] Optimizes Generates bytecode for ordering based on the given order

2018-11-15 Thread wenchen
Repository: spark Updated Branches: refs/heads/master d4130ec1f -> 44d4ef60b [SPARK-25974][SQL] Optimizes Generates bytecode for ordering based on the given order ## What changes were proposed in this pull request? Currently, when generates the code for ordering based on the given order,

spark git commit: [SPARK-26014][R] Deprecate R prior to version 3.4 in SparkR

2018-11-15 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 03306a6df -> d4130ec1f [SPARK-26014][R] Deprecate R prior to version 3.4 in SparkR ## What changes were proposed in this pull request? This PR proposes to bump up the minimum versions of R from 3.1 to 3.4. R version. 3.1.x is too old.