spark git commit: [SPARK-26105][PYTHON] Clean unittest2 imports up that were added for Python 2.6 before

Repository: spark Updated Branches: refs/heads/master 034ae305c -> bbbdaa82a [SPARK-26105][PYTHON] Clean unittest2 imports up that were added for Python 2.6 before ## What changes were proposed in this pull request? Currently, some of PySpark tests sill assume the tests could be ran in Pytho

spark git commit: [SPARK-24665][PYSPARK] Use SQLConf in PySpark to manage all sql configs

Repository: spark Updated Branches: refs/heads/master f825847c8 -> 8f91c697e [SPARK-24665][PYSPARK] Use SQLConf in PySpark to manage all sql configs ## What changes were proposed in this pull request? Use SQLConf for PySpark to manage all sql configs, drop all the hard code in config usage.

spark git commit: [SPARK-24715][BUILD] Override jline version as 2.14.3 in SBT

Repository: spark Updated Branches: refs/heads/master 8f91c697e -> 8008f9cb8 [SPARK-24715][BUILD] Override jline version as 2.14.3 in SBT ## What changes were proposed in this pull request? During SPARK-24418 (Upgrade Scala to 2.11.12 and 2.12.6), we upgrade `jline` version together. So, `mv

spark git commit: [SPARK-24507][DOCUMENTATION] Update streaming guide

Repository: spark Updated Branches: refs/heads/branch-2.3 3c0af793f -> 1cba0505e [SPARK-24507][DOCUMENTATION] Update streaming guide ## What changes were proposed in this pull request? Updated streaming guide for direct stream and link to integration guide. ## How was this patch tested? jekyl

spark git commit: [SPARK-24507][DOCUMENTATION] Update streaming guide

Repository: spark Updated Branches: refs/heads/master 8008f9cb8 -> f599cde69 [SPARK-24507][DOCUMENTATION] Update streaming guide ## What changes were proposed in this pull request? Updated streaming guide for direct stream and link to integration guide. ## How was this patch tested? jekyll bu

[02/14] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.2

http://git-wip-us.apache.org/repos/asf/spark-website/blob/6bbac496/site/docs/2.1.2/api/python/pyspark.streaming.html -- diff --git a/site/docs/2.1.2/api/python/pyspark.streaming.html b/site/docs/2.1.2/api/python/pyspark.streaming.

[04/14] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.2

http://git-wip-us.apache.org/repos/asf/spark-website/blob/6bbac496/site/docs/2.1.2/api/python/pyspark.mllib.html -- diff --git a/site/docs/2.1.2/api/python/pyspark.mllib.html b/site/docs/2.1.2/api/python/pyspark.mllib.html index 3

[13/14] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.2

http://git-wip-us.apache.org/repos/asf/spark-website/blob/6bbac496/site/docs/2.1.2/api/python/_modules/pyspark/ml/feature.html -- diff --git a/site/docs/2.1.2/api/python/_modules/pyspark/ml/feature.html b/site/docs/2.1.2/api/pytho

[09/14] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.2

http://git-wip-us.apache.org/repos/asf/spark-website/blob/6bbac496/site/docs/2.1.2/api/python/_modules/pyspark/mllib/regression.html -- diff --git a/site/docs/2.1.2/api/python/_modules/pyspark/mllib/regression.html b/site/docs/2.1

[06/14] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.2

http://git-wip-us.apache.org/repos/asf/spark-website/blob/6bbac496/site/docs/2.1.2/api/python/_modules/pyspark/streaming/dstream.html -- diff --git a/site/docs/2.1.2/api/python/_modules/pyspark/streaming/dstream.html b/site/docs/2

[01/14] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.2

Repository: spark-website Updated Branches: refs/heads/asf-site 775127770 -> 6bbac4966 http://git-wip-us.apache.org/repos/asf/spark-website/blob/6bbac496/site/docs/2.1.2/api/python/searchindex.js -- diff --git a/site/docs/2.1.2

[10/14] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.2

http://git-wip-us.apache.org/repos/asf/spark-website/blob/6bbac496/site/docs/2.1.2/api/python/_modules/pyspark/mllib/clustering.html -- diff --git a/site/docs/2.1.2/api/python/_modules/pyspark/mllib/clustering.html b/site/docs/2.1

[14/14] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.2

Fix signature description broken in PySpark API documentation in 2.1.2 Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/6bbac496 Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/6bbac496 Diff: htt

[08/14] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.2

http://git-wip-us.apache.org/repos/asf/spark-website/blob/6bbac496/site/docs/2.1.2/api/python/_modules/pyspark/serializers.html -- diff --git a/site/docs/2.1.2/api/python/_modules/pyspark/serializers.html b/site/docs/2.1.2/api/pyt

[11/14] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.2

http://git-wip-us.apache.org/repos/asf/spark-website/blob/6bbac496/site/docs/2.1.2/api/python/_modules/pyspark/ml/regression.html -- diff --git a/site/docs/2.1.2/api/python/_modules/pyspark/ml/regression.html b/site/docs/2.1.2/api

[05/14] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.2

http://git-wip-us.apache.org/repos/asf/spark-website/blob/6bbac496/site/docs/2.1.2/api/python/pyspark.ml.html -- diff --git a/site/docs/2.1.2/api/python/pyspark.ml.html b/site/docs/2.1.2/api/python/pyspark.ml.html index c7034f0..5

[03/14] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.2

http://git-wip-us.apache.org/repos/asf/spark-website/blob/6bbac496/site/docs/2.1.2/api/python/pyspark.sql.html -- diff --git a/site/docs/2.1.2/api/python/pyspark.sql.html b/site/docs/2.1.2/api/python/pyspark.sql.html index e2fbad9

[12/14] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.2

http://git-wip-us.apache.org/repos/asf/spark-website/blob/6bbac496/site/docs/2.1.2/api/python/_modules/pyspark/ml/param/shared.html -- diff --git a/site/docs/2.1.2/api/python/_modules/pyspark/ml/param/shared.html b/site/docs/2.1.2

[07/14] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.2

http://git-wip-us.apache.org/repos/asf/spark-website/blob/6bbac496/site/docs/2.1.2/api/python/_modules/pyspark/sql/functions.html -- diff --git a/site/docs/2.1.2/api/python/_modules/pyspark/sql/functions.html b/site/docs/2.1.2/api

[4/6] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.3

http://git-wip-us.apache.org/repos/asf/spark-website/blob/da71a5c1/site/docs/2.1.3/api/python/pyspark.mllib.html -- diff --git a/site/docs/2.1.3/api/python/pyspark.mllib.html b/site/docs/2.1.3/api/python/pyspark.mllib.html index 7

[1/6] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.3

Repository: spark-website Updated Branches: refs/heads/asf-site 6bbac4966 -> da71a5c1d http://git-wip-us.apache.org/repos/asf/spark-website/blob/da71a5c1/site/docs/2.1.3/api/python/searchindex.js -- diff --git a/site/docs/2.1.3

[3/6] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.3

http://git-wip-us.apache.org/repos/asf/spark-website/blob/da71a5c1/site/docs/2.1.3/api/python/pyspark.sql.html -- diff --git a/site/docs/2.1.3/api/python/pyspark.sql.html b/site/docs/2.1.3/api/python/pyspark.sql.html index 329ea36

[2/6] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.3

http://git-wip-us.apache.org/repos/asf/spark-website/blob/da71a5c1/site/docs/2.1.3/api/python/pyspark.streaming.html -- diff --git a/site/docs/2.1.3/api/python/pyspark.streaming.html b/site/docs/2.1.3/api/python/pyspark.streaming.

[6/6] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.3

Fix signature description broken in PySpark API documentation in 2.1.3 Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/da71a5c1 Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/da71a5c1 Diff: htt

[5/6] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.1.3

http://git-wip-us.apache.org/repos/asf/spark-website/blob/da71a5c1/site/docs/2.1.3/api/python/pyspark.ml.html -- diff --git a/site/docs/2.1.3/api/python/pyspark.ml.html b/site/docs/2.1.3/api/python/pyspark.ml.html index f37f2df..2

spark-website git commit: Fix nit in 2-1-3 release page

Repository: spark-website Updated Branches: refs/heads/asf-site da71a5c1d -> 8857572df Fix nit in 2-1-3 release page Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/8857572d Tree: http://git-wip-us.apache.o

spark git commit: [SPARK-24709][SQL] schema_of_json() - schema inference from an example

Repository: spark Updated Branches: refs/heads/master 5585c5765 -> 776f299fc [SPARK-24709][SQL] schema_of_json() - schema inference from an example ## What changes were proposed in this pull request? In the PR, I propose to add new function - *schema_of_json()* which infers schema of JSON st

spark git commit: [SPARK-23698] Remove raw_input() from Python 2

Repository: spark Updated Branches: refs/heads/master 776f299fc -> b42fda8ab [SPARK-23698] Remove raw_input() from Python 2 Signed-off-by: cclauss ## What changes were proposed in this pull request? Humans will be able to enter text in Python 3 prompts which they can not do today. The Pyth

spark git commit: [BUILD] Close stale PRs

Repository: spark Updated Branches: refs/heads/master b42fda8ab -> 5bf95f2a3 [BUILD] Close stale PRs Closes #20932 Closes #17843 Closes #13477 Closes #14291 Closes #20919 Closes #17907 Closes #18766 Closes #20809 Closes #8849 Closes #21076 Closes #21507 Closes #21336 Closes #21681 Closes #2169

spark git commit: [SPARK-24732][SQL] Type coercion between MapTypes.

Repository: spark Updated Branches: refs/heads/master 5bf95f2a3 -> 7c08eb6d6 [SPARK-24732][SQL] Type coercion between MapTypes. ## What changes were proposed in this pull request? Currently we don't allow type coercion between maps. We can support type coercion between MapTypes where both the

[3/7] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.2.1

http://git-wip-us.apache.org/repos/asf/spark-website/blob/26b52712/site/docs/2.2.1/api/python/pyspark.sql.html -- diff --git a/site/docs/2.2.1/api/python/pyspark.sql.html b/site/docs/2.2.1/api/python/pyspark.sql.html index 8b349cc

[1/7] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.2.1

Repository: spark-website Updated Branches: refs/heads/asf-site 8857572df -> 26b527127 http://git-wip-us.apache.org/repos/asf/spark-website/blob/26b52712/site/docs/2.2.1/api/python/searchindex.js -- diff --git a/site/docs/2.2.1

[6/7] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.2.1

http://git-wip-us.apache.org/repos/asf/spark-website/blob/26b52712/site/docs/2.2.1/api/python/_modules/pyspark/rdd.html -- diff --git a/site/docs/2.2.1/api/python/_modules/pyspark/rdd.html b/site/docs/2.2.1/api/python/_modules/pys

[2/7] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.2.1

http://git-wip-us.apache.org/repos/asf/spark-website/blob/26b52712/site/docs/2.2.1/api/python/pyspark.streaming.html -- diff --git a/site/docs/2.2.1/api/python/pyspark.streaming.html b/site/docs/2.2.1/api/python/pyspark.streaming.

[4/7] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.2.1

http://git-wip-us.apache.org/repos/asf/spark-website/blob/26b52712/site/docs/2.2.1/api/python/pyspark.mllib.html -- diff --git a/site/docs/2.2.1/api/python/pyspark.mllib.html b/site/docs/2.2.1/api/python/pyspark.mllib.html index c

[7/7] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.2.1

Fix signature description broken in PySpark API documentation in 2.2.1 Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/26b52712 Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/26b52712 Diff: htt

[5/7] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.2.1

http://git-wip-us.apache.org/repos/asf/spark-website/blob/26b52712/site/docs/2.2.1/api/python/pyspark.ml.html -- diff --git a/site/docs/2.2.1/api/python/pyspark.ml.html b/site/docs/2.2.1/api/python/pyspark.ml.html index 1398703..a

[4/7] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.3.1

http://git-wip-us.apache.org/repos/asf/spark-website/blob/5660fb9a/site/docs/2.3.1/api/python/pyspark.mllib.html -- diff --git a/site/docs/2.3.1/api/python/pyspark.mllib.html b/site/docs/2.3.1/api/python/pyspark.mllib.html index c

[7/7] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.3.1

Fix signature description broken in PySpark API documentation in 2.3.1 Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/5660fb9a Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/5660fb9a Diff: htt

[1/7] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.3.1

Repository: spark-website Updated Branches: refs/heads/asf-site 26b527127 -> 5660fb9a4 http://git-wip-us.apache.org/repos/asf/spark-website/blob/5660fb9a/site/docs/2.3.1/api/python/searchindex.js -- diff --git a/site/docs/2.3.1

[3/7] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.3.1

http://git-wip-us.apache.org/repos/asf/spark-website/blob/5660fb9a/site/docs/2.3.1/api/python/pyspark.sql.html -- diff --git a/site/docs/2.3.1/api/python/pyspark.sql.html b/site/docs/2.3.1/api/python/pyspark.sql.html index 43c51be

[2/7] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.3.1

http://git-wip-us.apache.org/repos/asf/spark-website/blob/5660fb9a/site/docs/2.3.1/api/python/pyspark.streaming.html -- diff --git a/site/docs/2.3.1/api/python/pyspark.streaming.html b/site/docs/2.3.1/api/python/pyspark.streaming.

[6/7] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.3.1

http://git-wip-us.apache.org/repos/asf/spark-website/blob/5660fb9a/site/docs/2.3.1/api/python/_modules/pyspark/profiler.html -- diff --git a/site/docs/2.3.1/api/python/_modules/pyspark/profiler.html b/site/docs/2.3.1/api/python/_m

[5/7] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.3.1

http://git-wip-us.apache.org/repos/asf/spark-website/blob/5660fb9a/site/docs/2.3.1/api/python/pyspark.ml.html -- diff --git a/site/docs/2.3.1/api/python/pyspark.ml.html b/site/docs/2.3.1/api/python/pyspark.ml.html index 4ada723..9

spark git commit: [SPARK-17213][SPARK-17213][FOLLOW-UP] Improve the test of

Repository: spark Updated Branches: refs/heads/master bf764a33b -> 489a5294d [SPARK-17213][SPARK-17213][FOLLOW-UP] Improve the test of ## What changes were proposed in this pull request? This is a minor improvement for the test of SPARK-17213 ## How was this patch tested? N/A Author: Xiao Li

spark git commit: [SPARK-24698][PYTHON] Fixed typo in pyspark.ml's Identifiable class.

Repository: spark Updated Branches: refs/heads/master 489a5294d -> f997be0c3 [SPARK-24698][PYTHON] Fixed typo in pyspark.ml's Identifiable class. ## What changes were proposed in this pull request? Fixed a small typo in the code that caused 20 random characters to be added to the UID, rather

spark git commit: [SPARK-24673][SQL] scala sql function from_utc_timestamp second argument could be Column instead of String

Repository: spark Updated Branches: refs/heads/master f997be0c3 -> 4be9f0c02 [SPARK-24673][SQL] scala sql function from_utc_timestamp second argument could be Column instead of String ## What changes were proposed in this pull request? Add an overloaded version to `from_utc_timestamp` and `t

spark git commit: [SPARK-24737][SQL] Type coercion between StructTypes.

Repository: spark Updated Branches: refs/heads/master e71e93aaa -> 01fcba2c6 [SPARK-24737][SQL] Type coercion between StructTypes. ## What changes were proposed in this pull request? We can support type coercion between `StructType`s where all the internal types are compatible. ## How was t

spark git commit: [SPARK-24692][TESTS] Improvement FilterPushdownBenchmark

Repository: spark Updated Branches: refs/heads/master 01fcba2c6 -> bf67f70c4 [SPARK-24692][TESTS] Improvement FilterPushdownBenchmark ## What changes were proposed in this pull request? Refer to the [`WideSchemaBenchmark`](https://github.com/apache/spark/blob/v2.3.1/sql/core/src/test/scala/or

spark git commit: [SPARK-24673][SQL][PYTHON][FOLLOWUP] Support Column arguments in timezone of from_utc_timestamp/to_utc_timestamp

Repository: spark Updated Branches: refs/heads/master 141953f4c -> a381bce72 [SPARK-24673][SQL][PYTHON][FOLLOWUP] Support Column arguments in timezone of from_utc_timestamp/to_utc_timestamp ## What changes were proposed in this pull request? This pr supported column arguments in timezone of

spark git commit: [SPARK-24749][SQL] Use sameType to compare Array's element type in ArrayContains

Repository: spark Updated Branches: refs/heads/master 4de0425df -> fc43690d3 [SPARK-24749][SQL] Use sameType to compare Array's element type in ArrayContains ## What changes were proposed in this pull request? We should use `DataType.sameType` to compare element type in `ArrayContains`, othe

spark git commit: [SPARK-24739][PYTHON] Make PySpark compatible with Python 3.7

Repository: spark Updated Branches: refs/heads/branch-2.3 e5cc5f699 -> 64c72b4de [SPARK-24739][PYTHON] Make PySpark compatible with Python 3.7 ## What changes were proposed in this pull request? This PR proposes to make PySpark compatible with Python 3.7. There are rather radical change in

spark git commit: [SPARK-24739][PYTHON] Make PySpark compatible with Python 3.7

Repository: spark Updated Branches: refs/heads/master fc43690d3 -> 74f6a92fc [SPARK-24739][PYTHON] Make PySpark compatible with Python 3.7 ## What changes were proposed in this pull request? This PR proposes to make PySpark compatible with Python 3.7. There are rather radical change in sema

spark git commit: [SPARK-24740][PYTHON][ML] Make PySpark's tests compatible with NumPy 1.14+

Repository: spark Updated Branches: refs/heads/master 74f6a92fc -> 044b33b2e [SPARK-24740][PYTHON][ML] Make PySpark's tests compatible with NumPy 1.14+ ## What changes were proposed in this pull request? This PR proposes to make PySpark's tests compatible with NumPy 0.14+ NumPy 0.14.x introdu

spark git commit: [SPARK-24268][SQL] Use datatype.simpleString in error messages

Repository: spark Updated Branches: refs/heads/master 034913b62 -> 1bd3d61f4 [SPARK-24268][SQL] Use datatype.simpleString in error messages ## What changes were proposed in this pull request? SPARK-22893 tried to unify error messages about dataTypes. Unfortunately, still many places were mis

[5/6] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.2.2

http://git-wip-us.apache.org/repos/asf/spark-website/blob/7b3e459e/site/docs/2.2.2/api/python/pyspark.ml.html -- diff --git a/site/docs/2.2.2/api/python/pyspark.ml.html b/site/docs/2.2.2/api/python/pyspark.ml.html index 1ba048c..d

[6/6] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.2.2

Fix signature description broken in PySpark API documentation in 2.2.2 Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/7b3e459e Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/7b3e459e Diff: htt

[2/6] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.2.2

http://git-wip-us.apache.org/repos/asf/spark-website/blob/7b3e459e/site/docs/2.2.2/api/python/pyspark.streaming.html -- diff --git a/site/docs/2.2.2/api/python/pyspark.streaming.html b/site/docs/2.2.2/api/python/pyspark.streaming.

[3/6] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.2.2

http://git-wip-us.apache.org/repos/asf/spark-website/blob/7b3e459e/site/docs/2.2.2/api/python/pyspark.sql.html -- diff --git a/site/docs/2.2.2/api/python/pyspark.sql.html b/site/docs/2.2.2/api/python/pyspark.sql.html index ef4555b

[1/6] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.2.2

Repository: spark-website Updated Branches: refs/heads/asf-site 2b5ba2f62 -> 7b3e459e2 http://git-wip-us.apache.org/repos/asf/spark-website/blob/7b3e459e/site/docs/2.2.2/api/python/searchindex.js -- diff --git a/site/docs/2.2.2

[4/6] spark-website git commit: Fix signature description broken in PySpark API documentation in 2.2.2

http://git-wip-us.apache.org/repos/asf/spark-website/blob/7b3e459e/site/docs/2.2.2/api/python/pyspark.mllib.html -- diff --git a/site/docs/2.2.2/api/python/pyspark.mllib.html b/site/docs/2.2.2/api/python/pyspark.mllib.html index b

spark git commit: [MINOR] Add Sphinx into dev/requirements.txt

Repository: spark Updated Branches: refs/heads/master eb6e98803 -> 4984f1af7 [MINOR] Add Sphinx into dev/requirements.txt ## What changes were proposed in this pull request? Not a big deal but this PR adds `sphinx` into `dev/requirements.txt` since we found it needed - https://github.com/ap

spark git commit: [SPARK-24530][PYTHON] Add a control to force Python version in Sphinx via environment variable, SPHINXPYTHON

Repository: spark Updated Branches: refs/heads/master 6078b891d -> 1f94bf492 [SPARK-24530][PYTHON] Add a control to force Python version in Sphinx via environment variable, SPHINXPYTHON ## What changes were proposed in this pull request? This PR proposes to add `SPHINXPYTHON` environment var

spark git commit: [SPARK-24530][PYTHON] Add a control to force Python version in Sphinx via environment variable, SPHINXPYTHON

Repository: spark Updated Branches: refs/heads/branch-2.3 72eb97ce9 -> 19542f5de [SPARK-24530][PYTHON] Add a control to force Python version in Sphinx via environment variable, SPHINXPYTHON ## What changes were proposed in this pull request? This PR proposes to add `SPHINXPYTHON` environment

spark-website git commit: Update release process to use Python 3 for Python API documentation

Repository: spark-website Updated Branches: refs/heads/asf-site 7b3e459e2 -> a6788714a Update release process to use Python 3 for Python API documentation Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/a67

spark git commit: [SPARK-24529][BUILD][TEST-MAVEN] Add spotbugs into maven build process

Repository: spark Updated Branches: refs/heads/master 3ab48f985 -> 5ad4735bd [SPARK-24529][BUILD][TEST-MAVEN] Add spotbugs into maven build process ## What changes were proposed in this pull request? This PR enables a Java bytecode check tool [spotbugs](https://spotbugs.github.io/) to avoid

spark git commit: [SPARK-24537][R] Add array_remove / array_zip / map_from_arrays / array_distinct

Repository: spark Updated Branches: refs/heads/master 75725057b -> e0f4f206b [SPARK-24537][R] Add array_remove / array_zip / map_from_arrays / array_distinct ## What changes were proposed in this pull request? Add array_remove / array_zip / map_from_arrays / array_distinct functions in SparkR

spark git commit: [SPARK-17091][SQL] Add rule to convert IN predicate to equivalent Parquet filter

Repository: spark Updated Branches: refs/heads/master f1a99ad58 -> e1de34113 [SPARK-17091][SQL] Add rule to convert IN predicate to equivalent Parquet filter ## What changes were proposed in this pull request? The original pr is: https://github.com/apache/spark/pull/18424 Add a new optimizer

spark git commit: [SPARK-24718][SQL] Timestamp support pushdown to parquet data source

Repository: spark Updated Branches: refs/heads/master 8aceb961c -> 43e4e851b [SPARK-24718][SQL] Timestamp support pushdown to parquet data source ## What changes were proposed in this pull request? `Timestamp` support pushdown to parquet data source. Only `TIMESTAMP_MICROS` and `TIMESTAMP_MIL

spark git commit: [SPARK-24813][TESTS][HIVE][HOTFIX] HiveExternalCatalogVersionsSuite still flaky; fall back to Apache archive

Repository: spark Updated Branches: refs/heads/master 5d62a985d -> bbc2ffc8a [SPARK-24813][TESTS][HIVE][HOTFIX] HiveExternalCatalogVersionsSuite still flaky; fall back to Apache archive ## What changes were proposed in this pull request? Try only unique ASF mirrors to download Spark release;

spark git commit: [SPARK-24813][TESTS][HIVE][HOTFIX] HiveExternalCatalogVersionsSuite still flaky; fall back to Apache archive

Repository: spark Updated Branches: refs/heads/branch-2.3 f9a2b0a87 -> dae352a29 [SPARK-24813][TESTS][HIVE][HOTFIX] HiveExternalCatalogVersionsSuite still flaky; fall back to Apache archive ## What changes were proposed in this pull request? Try only unique ASF mirrors to download Spark rele

spark git commit: [SPARK-24813][TESTS][HIVE][HOTFIX][BRANCH-2.2] HiveExternalCatalogVersionsSuite still flaky; fall back to Apache archive

Repository: spark Updated Branches: refs/heads/branch-2.2 a8537a5ab -> 4bc4ccd63 [SPARK-24813][TESTS][HIVE][HOTFIX][BRANCH-2.2] HiveExternalCatalogVersionsSuite still flaky; fall back to Apache archive ## What changes were proposed in this pull request? Try only unique ASF mirrors to downloa

spark git commit: [SPARK-23259][SQL] Clean up legacy code around hive external catalog and HiveClientImpl

Repository: spark Updated Branches: refs/heads/master 0f0d1865f -> d57a267b7 [SPARK-23259][SQL] Clean up legacy code around hive external catalog and HiveClientImpl ## What changes were proposed in this pull request? Three legacy statements are removed by this patch: - in HiveExternalCatalo

spark git commit: [SPARK-20220][DOCS] Documentation Add thrift scheduling pool config to scheduling docs

Repository: spark Updated Branches: refs/heads/master d57a267b7 -> f876d3fa8 [SPARK-20220][DOCS] Documentation Add thrift scheduling pool config to scheduling docs ## What changes were proposed in this pull request? The thrift scheduling pool configuration was removed from a previous release

spark git commit: Revert "[SPARK-24402][SQL] Optimize `In` expression when only one element in the collection or collection is empty"

Repository: spark Updated Branches: refs/heads/master f876d3fa8 -> 0ca16f6e1 Revert "[SPARK-24402][SQL] Optimize `In` expression when only one element in the collection or collection is empty" This reverts commit 0f0d1865f581a9158d73505471953656b173beba. Project: http://git-wip-us.apache.or

spark git commit: [SPARK-24529][BUILD][TEST-MAVEN][FOLLOW-UP] Set spotbugs-maven-plugin's fork to true

Repository: spark Updated Branches: refs/heads/master 681845fd6 -> fc2e18963 [SPARK-24529][BUILD][TEST-MAVEN][FOLLOW-UP] Set spotbugs-maven-plugin's fork to true ## What changes were proposed in this pull request? Set `spotbugs-maven-plugin`'s fork to `true`, otherwise will throw exception

spark git commit: [SPARK-24386][SPARK-24768][BUILD][FOLLOWUP] Fix lint-java and Scala 2.12 build.

Repository: spark Updated Branches: refs/heads/master 3b59d326c -> 34cb3b54e [SPARK-24386][SPARK-24768][BUILD][FOLLOWUP] Fix lint-java and Scala 2.12 build. ## What changes were proposed in this pull request? This pr fixes lint-java and Scala 2.12 build. lint-java: ``` [ERROR] src/test/reso

spark git commit: [SPARK-24854][SQL] Gathering all Avro options into the AvroOptions class

Repository: spark Updated Branches: refs/heads/master 753f11516 -> cd5d93c0e [SPARK-24854][SQL] Gathering all Avro options into the AvroOptions class ## What changes were proposed in this pull request? In the PR, I propose to put all `Avro` options in new class `AvroOptions` in the same way

spark git commit: [INFRA] Close stale PR

Repository: spark Updated Branches: refs/heads/master cd5d93c0e -> 1a4fda886 [INFRA] Close stale PR Closes #17422 Closes #17619 Closes #18034 Closes #18229 Closes #18268 Closes #17973 Closes #18125 Closes #18918 Closes #19274 Closes #19456 Closes #19510 Closes #19420 Closes #20090 Closes #2017

spark git commit: [SPARK-24858][SQL] Avoid unnecessary parquet footer reads

Repository: spark Updated Branches: refs/heads/master 8b7d4f842 -> 6a9a058e0 [SPARK-24858][SQL] Avoid unnecessary parquet footer reads ## What changes were proposed in this pull request? Currently the same Parquet footer is read twice in the function `buildReaderWithPartitionValues` of Parqu

spark git commit: [SPARK-24868][PYTHON] add sequence function in Python

Repository: spark Updated Branches: refs/heads/master 2b91d9918 -> 0ab07b357 [SPARK-24868][PYTHON] add sequence function in Python ## What changes were proposed in this pull request? Add ```sequence``` in functions.py ## How was this patch tested? Add doctest. Author: Huaxin Gao Closes #

spark git commit: [SPARK-24873][YARN] Turn off spark-shell noisy log output

Repository: spark Updated Branches: refs/heads/master 106880edc -> d7ae4247e [SPARK-24873][YARN] Turn off spark-shell noisy log output ## What changes were proposed in this pull request? [SPARK-24182](https://github.com/apache/spark/pull/21243) changed the `logApplicationReport` from `false`

spark git commit: [SPARK-24883][SQL] Avro: remove implicit class AvroDataFrameWriter/AvroDataFrameReader

Repository: spark Updated Branches: refs/heads/master 8817c68f5 -> f59de52a2 [SPARK-24883][SQL] Avro: remove implicit class AvroDataFrameWriter/AvroDataFrameReader ## What changes were proposed in this pull request? As per Reynold's comment: https://github.com/apache/spark/pull/21742#discus

spark git commit: [SQL][HIVE] Correct an assert message in function makeRDDForTable

Repository: spark Updated Branches: refs/heads/master f59de52a2 -> ab18b02e6 [SQL][HIVE] Correct an assert message in function makeRDDForTable ## What changes were proposed in this pull request? according to the context, "makeRDDForTablePartitions" in assert message should be "makeRDDForParti

spark git commit: [SQL][HIVE] Correct an assert message in function makeRDDForTable

Repository: spark Updated Branches: refs/heads/branch-2.3 bd6bfacb2 -> f5bc94861 [SQL][HIVE] Correct an assert message in function makeRDDForTable ## What changes were proposed in this pull request? according to the context, "makeRDDForTablePartitions" in assert message should be "makeRDDForP

spark git commit: [SPARK-22499][FOLLOWUP][SQL] Reduce input string expressions for Least and Greatest to reduce time in its test

Repository: spark Updated Branches: refs/heads/master 13a67b070 -> 3d5c61e5f [SPARK-22499][FOLLOWUP][SQL] Reduce input string expressions for Least and Greatest to reduce time in its test ## What changes were proposed in this pull request? It's minor and trivial but looks 2000 input is good

spark git commit: [SPARK-22499][FOLLOWUP][SQL] Reduce input string expressions for Least and Greatest to reduce time in its test

Repository: spark Updated Branches: refs/heads/branch-2.3 f5bc94861 -> 740a23d7d [SPARK-22499][FOLLOWUP][SQL] Reduce input string expressions for Least and Greatest to reduce time in its test ## What changes were proposed in this pull request? It's minor and trivial but looks 2000 input is g

spark git commit: [SPARK-22499][FOLLOWUP][SQL] Reduce input string expressions for Least and Greatest to reduce time in its test

Repository: spark Updated Branches: refs/heads/branch-2.2 144426cff -> f339e2fd7 [SPARK-22499][FOLLOWUP][SQL] Reduce input string expressions for Least and Greatest to reduce time in its test ## What changes were proposed in this pull request? It's minor and trivial but looks 2000 input is g

spark git commit: [SPARK-19018][SQL] Add support for custom encoding on csv writer

Repository: spark Updated Branches: refs/heads/master afb062753 -> 78e0a725e [SPARK-19018][SQL] Add support for custom encoding on csv writer ## What changes were proposed in this pull request? Add support for custom encoding on csv writer, see https://issues.apache.org/jira/browse/SPARK-190

spark git commit: [SPARK-24891][FOLLOWUP][HOT-FIX][2.3] Fix the Compilation Errors

Repository: spark Updated Branches: refs/heads/branch-2.3 6a5999286 -> 740606eb8 [SPARK-24891][FOLLOWUP][HOT-FIX][2.3] Fix the Compilation Errors ## What changes were proposed in this pull request? This PR is to fix the compilation failure in 2.3 build. https://amplab.cs.berkeley.edu/jenkins

spark git commit: [SPARK-24924][SQL] Add mapping for built-in Avro data source

Repository: spark Updated Branches: refs/heads/master c9b233d41 -> 58353d7f4 [SPARK-24924][SQL] Add mapping for built-in Avro data source ## What changes were proposed in this pull request? This PR aims to the followings. 1. Like `com.databricks.spark.csv` mapping, we had better map `com.dat

spark git commit: [SPARK-24829][STS] In Spark Thrift Server, CAST AS FLOAT inconsistent with spark-shell or spark-sql

Repository: spark Updated Branches: refs/heads/master 094aa5971 -> dc3713cca [SPARK-24829][STS] In Spark Thrift Server, CAST AS FLOAT inconsistent with spark-shell or spark-sql ## What changes were proposed in this pull request? SELECT CAST('4.56' AS FLOAT) the result is 4.55942779541

spark git commit: [SPARK-24929][INFRA] Make merge script don't swallow KeyboardInterrupt

Repository: spark Updated Branches: refs/heads/master dc3713cca -> f9c9d80e4 [SPARK-24929][INFRA] Make merge script don't swallow KeyboardInterrupt ## What changes were proposed in this pull request? If you want to get out of the loop to assign JIRA's user by command+c (KeyboardInterrupt), I

spark git commit: [SPARK-24881][SQL] New Avro option - compression

Repository: spark Updated Branches: refs/heads/master c9bec1d37 -> 0a0f68bae [SPARK-24881][SQL] New Avro option - compression ## What changes were proposed in this pull request? In the PR, I added new option for Avro datasource - `compression`. The option allows to specify compression codec

spark git commit: [SPARK-24624][SQL][PYTHON] Support mixture of Python UDF and Scalar Pandas UDF

Repository: spark Updated Branches: refs/heads/master 6424b146c -> e8752095a [SPARK-24624][SQL][PYTHON] Support mixture of Python UDF and Scalar Pandas UDF ## What changes were proposed in this pull request? This PR add supports for using mixed Python UDF and Scalar Pandas UDF, in the follow

spark git commit: [SPARK-24924][SQL][FOLLOW-UP] Add mapping for built-in Avro data source

Repository: spark Updated Branches: refs/heads/master e8752095a -> c6a3db2fb [SPARK-24924][SQL][FOLLOW-UP] Add mapping for built-in Avro data source ## What changes were proposed in this pull request? Add one more test case for `com.databricks.spark.avro`. ## How was this patch tested? N/A A

spark git commit: [MINOR][CORE][TEST] Fix afterEach() in TastSetManagerSuite and TaskSchedulerImplSuite

Repository: spark Updated Branches: refs/heads/master 2c54aae1b -> 3695ba577 [MINOR][CORE][TEST] Fix afterEach() in TastSetManagerSuite and TaskSchedulerImplSuite ## What changes were proposed in this pull request? In the `afterEach()` method of both `TastSetManagerSuite` and `TaskScheduler

spark git commit: [MINOR][CORE][TEST] Fix afterEach() in TastSetManagerSuite and TaskSchedulerImplSuite

Repository: spark Updated Branches: refs/heads/branch-2.2 f52d0c451 -> c4b37696f [MINOR][CORE][TEST] Fix afterEach() in TastSetManagerSuite and TaskSchedulerImplSuite ## What changes were proposed in this pull request? In the `afterEach()` method of both `TastSetManagerSuite` and `TaskSched

spark git commit: [MINOR][CORE][TEST] Fix afterEach() in TastSetManagerSuite and TaskSchedulerImplSuite

Repository: spark Updated Branches: refs/heads/branch-2.3 71eb7d468 -> bad56bb7b [MINOR][CORE][TEST] Fix afterEach() in TastSetManagerSuite and TaskSchedulerImplSuite ## What changes were proposed in this pull request? In the `afterEach()` method of both `TastSetManagerSuite` and `TaskSched

spark git commit: [MINOR][BUILD] Remove -Phive-thriftserver profile within appveyor.yml

Repository: spark Updated Branches: refs/heads/master 3695ba577 -> 3210121fe [MINOR][BUILD] Remove -Phive-thriftserver profile within appveyor.yml ## What changes were proposed in this pull request? This PR propose to remove `-Phive-thriftserver` profile which seems not affecting the SparkR

<    1   2   3   4   5   6   7   8   9   10   >