spark git commit: [SPARK-26079][SQL] Ensure listener event delivery in StreamingQueryListenersConfSuite.

2018-11-16 Thread gurwls223
Repository: spark Updated Branches: refs/heads/branch-2.4 c23b801d3 -> 33f55d445 [SPARK-26079][SQL] Ensure listener event delivery in StreamingQueryListenersConfSuite. Events are dispatched on a separate thread, so need to wait for them to be actually delivered before checking that the

spark git commit: [SPARK-26079][SQL] Ensure listener event delivery in StreamingQueryListenersConfSuite.

2018-11-16 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master d2792046a -> 23cd0e6e9 [SPARK-26079][SQL] Ensure listener event delivery in StreamingQueryListenersConfSuite. Events are dispatched on a separate thread, so need to wait for them to be actually delivered before checking that the listener

svn commit: r30939 - in /dev/spark/3.0.0-SNAPSHOT-2018_11_16_18_25-d279204-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-16 Thread pwendell
Author: pwendell Date: Sat Nov 17 02:38:20 2018 New Revision: 30939 Log: Apache Spark 3.0.0-SNAPSHOT-2018_11_16_18_25-d279204 docs [This commit notification would consist of 1755 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r30938 - in /dev/spark/2.4.1-SNAPSHOT-2018_11_16_16_22-c23b801-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-16 Thread pwendell
Author: pwendell Date: Sat Nov 17 00:37:32 2018 New Revision: 30938 Log: Apache Spark 2.4.1-SNAPSHOT-2018_11_16_16_22-c23b801 docs [This commit notification would consist of 1476 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-26095][BUILD] Disable parallelization in make-distibution.sh.

2018-11-16 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 058c4602b -> d2792046a [SPARK-26095][BUILD] Disable parallelization in make-distibution.sh. It makes the build slower, but at least it doesn't hang. Seems that maven-shade-plugin has some issue with parallelization. Closes #23061 from

spark git commit: [SPARK-26092][SS] Use CheckpointFileManager to write the streaming metadata file

2018-11-16 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-2.4 77c0629cb -> c23b801d3 [SPARK-26092][SS] Use CheckpointFileManager to write the streaming metadata file ## What changes were proposed in this pull request? Use CheckpointFileManager to write the streaming `metadata` file so that the

spark git commit: [SPARK-26092][SS] Use CheckpointFileManager to write the streaming metadata file

2018-11-16 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master 99cbc51b3 -> 058c4602b [SPARK-26092][SS] Use CheckpointFileManager to write the streaming metadata file ## What changes were proposed in this pull request? Use CheckpointFileManager to write the streaming `metadata` file so that the

svn commit: r30931 - in /dev/spark/2.4.1-SNAPSHOT-2018_11_16_12_15-77c0629-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-16 Thread pwendell
Author: pwendell Date: Fri Nov 16 20:30:12 2018 New Revision: 30931 Log: Apache Spark 2.4.1-SNAPSHOT-2018_11_16_12_15-77c0629 docs [This commit notification would consist of 1476 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r30929 - in /dev/spark/3.0.0-SNAPSHOT-2018_11_16_10_10-99cbc51-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-16 Thread pwendell
Author: pwendell Date: Fri Nov 16 18:22:40 2018 New Revision: 30929 Log: Apache Spark 3.0.0-SNAPSHOT-2018_11_16_10_10-99cbc51 docs [This commit notification would consist of 1755 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-26069][TESTS] Fix flaky test: RpcIntegrationSuite.sendRpcWithStreamFailures

2018-11-16 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-2.4 be42bfe5c -> 77c0629cb [SPARK-26069][TESTS] Fix flaky test: RpcIntegrationSuite.sendRpcWithStreamFailures ## What changes were proposed in this pull request? The test failure is because `assertErrorAndClosed` misses one possible

spark git commit: [SPARK-26069][TESTS] Fix flaky test: RpcIntegrationSuite.sendRpcWithStreamFailures

2018-11-16 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master a2fc48c28 -> 99cbc51b3 [SPARK-26069][TESTS] Fix flaky test: RpcIntegrationSuite.sendRpcWithStreamFailures ## What changes were proposed in this pull request? The test failure is because `assertErrorAndClosed` misses one possible error

svn commit: r30927 - in /dev/spark/2.4.1-SNAPSHOT-2018_11_16_08_06-be42bfe-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-16 Thread pwendell
Author: pwendell Date: Fri Nov 16 16:23:25 2018 New Revision: 30927 Log: Apache Spark 2.4.1-SNAPSHOT-2018_11_16_08_06-be42bfe docs [This commit notification would consist of 1476 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r30926 - in /dev/spark/2.3.3-SNAPSHOT-2018_11_16_08_06-550408e-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-16 Thread pwendell
Author: pwendell Date: Fri Nov 16 16:21:37 2018 New Revision: 30926 Log: Apache Spark 2.3.3-SNAPSHOT-2018_11_16_08_06-550408e docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

[2/2] spark git commit: [SPARK-26034][PYTHON][TESTS] Break large mllib/tests.py file into smaller files

2018-11-16 Thread gurwls223
[SPARK-26034][PYTHON][TESTS] Break large mllib/tests.py file into smaller files ## What changes were proposed in this pull request? This PR breaks down the large mllib/tests.py file that contains all Python MLlib unit tests into several smaller test files to be easier to read and maintain.

[1/2] spark git commit: [SPARK-26034][PYTHON][TESTS] Break large mllib/tests.py file into smaller files

2018-11-16 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 696b75a81 -> a2fc48c28 http://git-wip-us.apache.org/repos/asf/spark/blob/a2fc48c2/python/pyspark/mllib/tests/test_algorithms.py -- diff --git

spark git commit: [SPARK-25934][MESOS] Don't propagate SPARK_CONF_DIR from spark submit

2018-11-16 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.3 7a596187e -> 550408e56 [SPARK-25934][MESOS] Don't propagate SPARK_CONF_DIR from spark submit ## What changes were proposed in this pull request? Don't propagate SPARK_CONF_DIR to the driver in mesos cluster mode. ## How was this

spark git commit: [SPARK-25934][MESOS] Don't propagate SPARK_CONF_DIR from spark submit

2018-11-16 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.4 ed9fa790c -> be42bfe5c [SPARK-25934][MESOS] Don't propagate SPARK_CONF_DIR from spark submit ## What changes were proposed in this pull request? Don't propagate SPARK_CONF_DIR to the driver in mesos cluster mode. ## How was this

spark git commit: [SPARK-25934][MESOS] Don't propagate SPARK_CONF_DIR from spark submit

2018-11-16 Thread srowen
Repository: spark Updated Branches: refs/heads/master 2aef79a65 -> 696b75a81 [SPARK-25934][MESOS] Don't propagate SPARK_CONF_DIR from spark submit ## What changes were proposed in this pull request? Don't propagate SPARK_CONF_DIR to the driver in mesos cluster mode. ## How was this patch

spark git commit: [SPARK-25023] More detailed security guidance for K8S

2018-11-16 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.4 2d67be932 -> ed9fa790c [SPARK-25023] More detailed security guidance for K8S ## What changes were proposed in this pull request? Highlights specific security issues to be aware of with Spark on K8S and recommends K8S mechanisms that

spark git commit: [SPARK-25023] More detailed security guidance for K8S

2018-11-16 Thread srowen
Repository: spark Updated Branches: refs/heads/master 4ac8f9bec -> 2aef79a65 [SPARK-25023] More detailed security guidance for K8S ## What changes were proposed in this pull request? Highlights specific security issues to be aware of with Spark on K8S and recommends K8S mechanisms that