svn commit: r23839 - in /dev/spark/2.3.0-SNAPSHOT-2017_12_21_00_01-cb9fc8d-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2017-12-21 Thread pwendell
Author: pwendell Date: Thu Dec 21 08:14:49 2017 New Revision: 23839 Log: Apache Spark 2.3.0-SNAPSHOT-2017_12_21_00_01-cb9fc8d docs [This commit notification would consist of 1415 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

[2/2] spark git commit: [SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0

2017-12-21 Thread gurwls223
[SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0 ## What changes were proposed in this pull request? Upgrade Spark to Arrow 0.8.0 for Java and Python. Also includes an upgrade of Netty to 4.1.17 to resolve dependency requirements. The highlights that pertain to Spark for the update from Arrow

[1/2] spark git commit: [SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0

2017-12-21 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master cb9fc8d9b -> 59d52631e http://git-wip-us.apache.org/repos/asf/spark/blob/59d52631/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowConvertersSuite.scala --

svn commit: r23847 - in /dev/spark/2.3.0-SNAPSHOT-2017_12_21_04_01-59d5263-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2017-12-21 Thread pwendell
Author: pwendell Date: Thu Dec 21 12:16:56 2017 New Revision: 23847 Log: Apache Spark 2.3.0-SNAPSHOT-2017_12_21_04_01-59d5263 docs [This commit notification would consist of 1415 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-22852][BUILD] Exclude -Xlint:unchecked from sbt javadoc flags

2017-12-21 Thread srowen
Repository: spark Updated Branches: refs/heads/master 59d52631e -> 0abaf31be [SPARK-22852][BUILD] Exclude -Xlint:unchecked from sbt javadoc flags ## What changes were proposed in this pull request? Moves the -Xlint:unchecked flag in the sbt build configuration from Compile to (Compile, compi

spark git commit: [SPARK-22855][BUILD] Add -no-java-comments to sbt docs/scalacOptions

2017-12-21 Thread srowen
Repository: spark Updated Branches: refs/heads/master 0abaf31be -> 4c2efde93 [SPARK-22855][BUILD] Add -no-java-comments to sbt docs/scalacOptions Prevents Scala 2.12 scaladoc from blowing up attempting to parse java comments. ## What changes were proposed in this pull request? Adds -no-java-

svn commit: r23848 - in /dev/spark/2.3.0-SNAPSHOT-2017_12_21_08_01-0abaf31-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2017-12-21 Thread pwendell
Author: pwendell Date: Thu Dec 21 16:17:25 2017 New Revision: 23848 Log: Apache Spark 2.3.0-SNAPSHOT-2017_12_21_08_01-0abaf31 docs [This commit notification would consist of 1415 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-22668][SQL] Ensure no global variables in arguments of method split by CodegenContext.splitExpressions()

2017-12-21 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 4c2efde93 -> 8a0ed5a5e [SPARK-22668][SQL] Ensure no global variables in arguments of method split by CodegenContext.splitExpressions() ## What changes were proposed in this pull request? Passing global variables to the split method is dan

spark git commit: [SPARK-22786][SQL] only use AppStatusPlugin in history server

2017-12-21 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 8a0ed5a5e -> d3a1d9527 [SPARK-22786][SQL] only use AppStatusPlugin in history server ## What changes were proposed in this pull request? In https://github.com/apache/spark/pull/19681 we introduced a new interface called `AppStatusPlugin`,

[2/3] spark git commit: [SPARK-22822][TEST] Basic tests for WindowFrameCoercion and DecimalPrecision

2017-12-21 Thread lixiao
http://git-wip-us.apache.org/repos/asf/spark/blob/4e107fdb/sql/core/src/test/resources/sql-tests/results/typeCoercion/native/decimalPrecision.sql.out -- diff --git a/sql/core/src/test/resources/sql-tests/results/typeCoercion/nativ

[1/3] spark git commit: [SPARK-22822][TEST] Basic tests for WindowFrameCoercion and DecimalPrecision

2017-12-21 Thread lixiao
Repository: spark Updated Branches: refs/heads/master d3a1d9527 -> 4e107fdb7 http://git-wip-us.apache.org/repos/asf/spark/blob/4e107fdb/sql/core/src/test/resources/sql-tests/results/typeCoercion/native/windowFrameCoercion.sql.out -

[3/3] spark git commit: [SPARK-22822][TEST] Basic tests for WindowFrameCoercion and DecimalPrecision

2017-12-21 Thread lixiao
[SPARK-22822][TEST] Basic tests for WindowFrameCoercion and DecimalPrecision ## What changes were proposed in this pull request? Test Coverage for `WindowFrameCoercion` and `DecimalPrecision`, this is a Sub-tasks for [SPARK-22722](https://issues.apache.org/jira/browse/SPARK-22722). ## How was t

spark git commit: [SPARK-22042][FOLLOW-UP][SQL] ReorderJoinPredicates can break when child's partitioning is not decided

2017-12-21 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 4e107fdb7 -> fe65361b0 [SPARK-22042][FOLLOW-UP][SQL] ReorderJoinPredicates can break when child's partitioning is not decided ## What changes were proposed in this pull request? This is a followup PR of https://github.com/apache/spark/pul

svn commit: r23856 - in /dev/spark/2.3.0-SNAPSHOT-2017_12_21_12_01-fe65361-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2017-12-21 Thread pwendell
Author: pwendell Date: Thu Dec 21 20:14:34 2017 New Revision: 23856 Log: Apache Spark 2.3.0-SNAPSHOT-2017_12_21_12_01-fe65361 docs [This commit notification would consist of 1415 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-22861][SQL] SQLAppStatusListener handles multi-job executions.

2017-12-21 Thread vanzin
Repository: spark Updated Branches: refs/heads/master fe65361b0 -> 7beb375bf [SPARK-22861][SQL] SQLAppStatusListener handles multi-job executions. When one execution has multiple jobs, we need to append to the set of stages, not replace them on every job. Added unit test and ran existing test

svn commit: r23861 - in /dev/spark/2.3.0-SNAPSHOT-2017_12_21_16_01-7beb375-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2017-12-21 Thread pwendell
Author: pwendell Date: Fri Dec 22 00:14:35 2017 New Revision: 23861 Log: Apache Spark 2.3.0-SNAPSHOT-2017_12_21_16_01-7beb375 docs [This commit notification would consist of 1415 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-22648][K8S] Spark on Kubernetes - Documentation

2017-12-21 Thread rxin
Repository: spark Updated Branches: refs/heads/master 7beb375bf -> 7ab165b70 [SPARK-22648][K8S] Spark on Kubernetes - Documentation What changes were proposed in this pull request? This PR contains documentation on the usage of Kubernetes scheduler in Spark 2.3, and a shell script to make it

spark git commit: [SPARK-22854][UI] Read Spark version from event logs.

2017-12-21 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 7ab165b70 -> c0abb1d99 [SPARK-22854][UI] Read Spark version from event logs. The code was ignoring SparkListenerLogStart, which was added somewhat recently to record the Spark version used to generate an event log. Author: Marcelo Vanzin

spark git commit: [SPARK-22750][SQL] Reuse mutable states when possible

2017-12-21 Thread wenchen
Repository: spark Updated Branches: refs/heads/master c0abb1d99 -> c6f01cade [SPARK-22750][SQL] Reuse mutable states when possible ## What changes were proposed in this pull request? The PR introduces a new method `addImmutableStateIfNotExists ` to `CodeGenerator` to allow reusing and sharin

spark git commit: [SPARK-22450][CORE][MLLIB][FOLLOWUP] safely register class for mllib - LabeledPoint/VectorWithNorm/TreePoint

2017-12-21 Thread srowen
Repository: spark Updated Branches: refs/heads/master c6f01cade -> a36b78b0e [SPARK-22450][CORE][MLLIB][FOLLOWUP] safely register class for mllib - LabeledPoint/VectorWithNorm/TreePoint ## What changes were proposed in this pull request? register following classes in Kryo: `org.apache.spark.m

svn commit: r23862 - in /dev/spark/2.3.0-SNAPSHOT-2017_12_21_20_01-a36b78b-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2017-12-21 Thread pwendell
Author: pwendell Date: Fri Dec 22 04:14:43 2017 New Revision: 23862 Log: Apache Spark 2.3.0-SNAPSHOT-2017_12_21_20_01-a36b78b docs [This commit notification would consist of 1415 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-22866][K8S] Fix path issue in Kubernetes dockerfile

2017-12-21 Thread mridulm80
Repository: spark Updated Branches: refs/heads/master a36b78b0e -> 22e1849bc [SPARK-22866][K8S] Fix path issue in Kubernetes dockerfile ## What changes were proposed in this pull request? The path was recently changed in https://github.com/apache/spark/pull/19946, but the dockerfile was not

spark git commit: [SPARK-22862] Docs on lazy elimination of columns missing from an encoder

2017-12-21 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.2 1e4cca02f -> 1cf3e3a26 [SPARK-22862] Docs on lazy elimination of columns missing from an encoder This behavior has confused some users, so lets clarify it. Author: Michael Armbrust Closes #20048 from marmbrus/datasetAsDocs. (cherry

spark git commit: [SPARK-22862] Docs on lazy elimination of columns missing from an encoder

2017-12-21 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 22e1849bc -> 8df1da396 [SPARK-22862] Docs on lazy elimination of columns missing from an encoder This behavior has confused some users, so lets clarify it. Author: Michael Armbrust Closes #20048 from marmbrus/datasetAsDocs. Project: ht

spark git commit: [SPARK-22862] Docs on lazy elimination of columns missing from an encoder

2017-12-21 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.1 0f6862106 -> 1df8020e1 [SPARK-22862] Docs on lazy elimination of columns missing from an encoder This behavior has confused some users, so lets clarify it. Author: Michael Armbrust Closes #20048 from marmbrus/datasetAsDocs. (cherry