spark git commit: [SPARK-20889][SPARKR] Grouped documentation for AGGREGATE column methods

2017-06-19 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master 9b57cd8d5 -> 8965fe764 [SPARK-20889][SPARKR] Grouped documentation for AGGREGATE column methods ## What changes were proposed in this pull request? Grouped documentation for the aggregate functions for Column. Author: actuaryzhang

spark git commit: [SPARK-21133][CORE] Fix HighlyCompressedMapStatus#writeExternal throws NPE

2017-06-19 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 cf10fa88c -> 8bf7f1eba [SPARK-21133][CORE] Fix HighlyCompressedMapStatus#writeExternal throws NPE ## What changes were proposed in this pull request? Fix HighlyCompressedMapStatus#writeExternal NPE: ``` 17/06/18 15:00:27 ERROR Utils:

spark git commit: [SPARK-21133][CORE] Fix HighlyCompressedMapStatus#writeExternal throws NPE

2017-06-19 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 9eacc5e43 -> 9b57cd8d5 [SPARK-21133][CORE] Fix HighlyCompressedMapStatus#writeExternal throws NPE ## What changes were proposed in this pull request? Fix HighlyCompressedMapStatus#writeExternal NPE: ``` 17/06/18 15:00:27 ERROR Utils:

spark git commit: [INFRA] Close stale PRs.

2017-06-19 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 3d4d11a80 -> 9eacc5e43 [INFRA] Close stale PRs. Closes #18311 Closes #18278 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9eacc5e4 Tree:

spark git commit: [SPARK-21138][YARN] Cannot delete staging dir when the clusters of "spark.yarn.stagingDir" and "spark.hadoop.fs.defaultFS" are different

2017-06-19 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.0 333924e8f -> 44a97f70f [SPARK-21138][YARN] Cannot delete staging dir when the clusters of "spark.yarn.stagingDir" and "spark.hadoop.fs.defaultFS" are different ## What changes were proposed in this pull request? When I set different

spark git commit: [SPARK-21138][YARN] Cannot delete staging dir when the clusters of "spark.yarn.stagingDir" and "spark.hadoop.fs.defaultFS" are different

2017-06-19 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.1 a44c118e2 -> 7799f35dd [SPARK-21138][YARN] Cannot delete staging dir when the clusters of "spark.yarn.stagingDir" and "spark.hadoop.fs.defaultFS" are different ## What changes were proposed in this pull request? When I set different

spark git commit: [SPARK-21138][YARN] Cannot delete staging dir when the clusters of "spark.yarn.stagingDir" and "spark.hadoop.fs.defaultFS" are different

2017-06-19 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.2 e329beaff -> cf10fa88c [SPARK-21138][YARN] Cannot delete staging dir when the clusters of "spark.yarn.stagingDir" and "spark.hadoop.fs.defaultFS" are different ## What changes were proposed in this pull request? When I set different

spark git commit: [SPARK-21138][YARN] Cannot delete staging dir when the clusters of "spark.yarn.stagingDir" and "spark.hadoop.fs.defaultFS" are different

2017-06-19 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 581565dd8 -> 3d4d11a80 [SPARK-21138][YARN] Cannot delete staging dir when the clusters of "spark.yarn.stagingDir" and "spark.hadoop.fs.defaultFS" are different ## What changes were proposed in this pull request? When I set different

spark git commit: [SPARK-21124][UI] Show correct application user in UI.

2017-06-19 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 0a4b7e4f8 -> 581565dd8 [SPARK-21124][UI] Show correct application user in UI. The jobs page currently shows the application user, but it assumes the OS user is the same as the user running the application, which may not be true in all

spark git commit: [MINOR] Fix some typo of the document

2017-06-19 Thread srowen
Repository: spark Updated Branches: refs/heads/master ecc563135 -> 0a4b7e4f8 [MINOR] Fix some typo of the document ## What changes were proposed in this pull request? Fix some typo of the document. ## How was this patch tested? Existing tests. Please review

spark git commit: [MINOR][BUILD] Fix Java linter errors

2017-06-19 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 7b50736c4 -> e329beaff [MINOR][BUILD] Fix Java linter errors This PR cleans up a few Java linter errors for Apache Spark 2.2 release. ```bash $ dev/lint-java Using `mvn` from path: /usr/local/bin/mvn Checkstyle checks passed. ``` We

spark git commit: [MINOR][BUILD] Fix Java linter errors

2017-06-19 Thread srowen
Repository: spark Updated Branches: refs/heads/master e5387018e -> ecc563135 [MINOR][BUILD] Fix Java linter errors ## What changes were proposed in this pull request? This PR cleans up a few Java linter errors for Apache Spark 2.2 release. ## How was this patch tested? ```bash $

spark git commit: [SPARK-19975][PYTHON][SQL] Add map_keys and map_values functions to Python

2017-06-19 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 66a792cd8 -> e5387018e [SPARK-19975][PYTHON][SQL] Add map_keys and map_values functions to Python ## What changes were proposed in this pull request? This fix tries to address the issue in SPARK-19975 where we have `map_keys` and

spark git commit: [SPARK-21123][DOCS][STRUCTURED STREAMING] Options for file stream source are in a wrong table

2017-06-19 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-2.2 f7fcdec6c -> 7b50736c4 [SPARK-21123][DOCS][STRUCTURED STREAMING] Options for file stream source are in a wrong table ## What changes were proposed in this pull request? The description for several options of File Source for

spark git commit: [SPARK-21123][DOCS][STRUCTURED STREAMING] Options for file stream source are in a wrong table

2017-06-19 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master e92ffe6f1 -> 66a792cd8 [SPARK-21123][DOCS][STRUCTURED STREAMING] Options for file stream source are in a wrong table ## What changes were proposed in this pull request? The description for several options of File Source for structured

spark git commit: [SPARK-19688][STREAMING] Not to read `spark.yarn.credentials.file` from checkpoint.

2017-06-19 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.0 7efd475d4 -> 333924e8f [SPARK-19688][STREAMING] Not to read `spark.yarn.credentials.file` from checkpoint. ## What changes were proposed in this pull request? Reload the `spark.yarn.credentials.file` property when restarting a

spark git commit: [SPARK-19688][STREAMING] Not to read `spark.yarn.credentials.file` from checkpoint.

2017-06-19 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-1.6 6b315f3d5 -> a233fac0b [SPARK-19688][STREAMING] Not to read `spark.yarn.credentials.file` from checkpoint. ## What changes were proposed in this pull request? Reload the `spark.yarn.credentials.file` property when restarting a

spark git commit: [SPARK-19688][STREAMING] Not to read `spark.yarn.credentials.file` from checkpoint.

2017-06-19 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.1 0ebb3b843 -> a44c118e2 [SPARK-19688][STREAMING] Not to read `spark.yarn.credentials.file` from checkpoint. ## What changes were proposed in this pull request? Reload the `spark.yarn.credentials.file` property when restarting a

spark git commit: [SPARK-19688][STREAMING] Not to read `spark.yarn.credentials.file` from checkpoint.

2017-06-19 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.2 fab070ca4 -> f7fcdec6c [SPARK-19688][STREAMING] Not to read `spark.yarn.credentials.file` from checkpoint. ## What changes were proposed in this pull request? Reload the `spark.yarn.credentials.file` property when restarting a

spark git commit: [SPARK-19688][STREAMING] Not to read `spark.yarn.credentials.file` from checkpoint.

2017-06-19 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 9a145fd79 -> e92ffe6f1 [SPARK-19688][STREAMING] Not to read `spark.yarn.credentials.file` from checkpoint. ## What changes were proposed in this pull request? Reload the `spark.yarn.credentials.file` property when restarting a streaming

spark git commit: [MINOR] Bump SparkR and PySpark version to 2.3.0.

2017-06-19 Thread srowen
Repository: spark Updated Branches: refs/heads/master 9413b84b5 -> 9a145fd79 [MINOR] Bump SparkR and PySpark version to 2.3.0. ## What changes were proposed in this pull request? #17753 bumps master branch version to 2.3.0-SNAPSHOT, but it seems SparkR and PySpark version were omitted.

spark git commit: [SPARK-21132][SQL] DISTINCT modifier of function arguments should not be silently ignored

2017-06-19 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 d3c79b77a -> fab070ca4 [SPARK-21132][SQL] DISTINCT modifier of function arguments should not be silently ignored ### What changes were proposed in this pull request? We should not silently ignore `DISTINCT` when they are not supported

spark git commit: [SPARK-21132][SQL] DISTINCT modifier of function arguments should not be silently ignored

2017-06-19 Thread wenchen
Repository: spark Updated Branches: refs/heads/master ea542d29b -> 9413b84b5 [SPARK-21132][SQL] DISTINCT modifier of function arguments should not be silently ignored ### What changes were proposed in this pull request? We should not silently ignore `DISTINCT` when they are not supported in