spark git commit: [SPARK-21345][SQL][TEST][TEST-MAVEN][BRANCH-2.1] SparkSessionBuilderSuite should clean up stopped sessions.

2017-07-08 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.1 7f7b63bb6 -> 5e2bfd5bc [SPARK-21345][SQL][TEST][TEST-MAVEN][BRANCH-2.1] SparkSessionBuilderSuite should clean up stopped sessions. ## What changes were proposed in this pull request? `SparkSessionBuilderSuite` should clean up stopped

spark git commit: [SPARK-21093][R] Terminate R's worker processes in the parent of R's daemon to prevent a leak

2017-07-08 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master c3712b77a -> 08e0d033b [SPARK-21093][R] Terminate R's worker processes in the parent of R's daemon to prevent a leak ## What changes were proposed in this pull request? This is a retry for #18320. This PR was reverted due to unexpected

spark git commit: [SPARK-21307][REVERT][SQL] Remove SQLConf parameters from the parser-related classes

2017-07-08 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 062c336d0 -> c3712b77a [SPARK-21307][REVERT][SQL] Remove SQLConf parameters from the parser-related classes ## What changes were proposed in this pull request? Since we do not set active sessions when parsing the plan, we are unable to

spark git commit: [SPARK-21343] Refine the document for spark.reducer.maxReqSizeShuffleToMem.

2017-07-08 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 c8d7855b9 -> 964332b28 [SPARK-21343] Refine the document for spark.reducer.maxReqSizeShuffleToMem. ## What changes were proposed in this pull request? In current code, reducer can break the old shuffle service when

spark git commit: [SPARK-20342][CORE] Update task accumulators before sending task end event.

2017-07-08 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 a64f10800 -> c8d7855b9 [SPARK-20342][CORE] Update task accumulators before sending task end event. This makes sures that listeners get updated task information; otherwise it's possible to write incomplete task information into event

spark git commit: [SPARK-20342][CORE] Update task accumulators before sending task end event.

2017-07-08 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 9fccc3627 -> 9131bdb7e [SPARK-20342][CORE] Update task accumulators before sending task end event. This makes sures that listeners get updated task information; otherwise it's possible to write incomplete task information into event logs,

spark git commit: [SPARK-21083][SQL] Store zero size and row count when analyzing empty table

2017-07-08 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 0b8dd2d08 -> 9fccc3627 [SPARK-21083][SQL] Store zero size and row count when analyzing empty table ## What changes were proposed in this pull request? We should be able to store zero size and row count after analyzing empty table. This

spark git commit: [SPARK-21345][SQL][TEST][TEST-MAVEN] SparkSessionBuilderSuite should clean up stopped sessions.

2017-07-08 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 7d0b1c927 -> a64f10800 [SPARK-21345][SQL][TEST][TEST-MAVEN] SparkSessionBuilderSuite should clean up stopped sessions. `SparkSessionBuilderSuite` should clean up stopped sessions. Otherwise, it leaves behind some stopped

spark git commit: [SPARK-21345][SQL][TEST][TEST-MAVEN] SparkSessionBuilderSuite should clean up stopped sessions.

2017-07-08 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 330bf5c99 -> 0b8dd2d08 [SPARK-21345][SQL][TEST][TEST-MAVEN] SparkSessionBuilderSuite should clean up stopped sessions. ## What changes were proposed in this pull request? `SparkSessionBuilderSuite` should clean up stopped sessions.

spark git commit: [SPARK-21228][SQL][BRANCH-2.2] InSet incorrect handling of structs

2017-07-08 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 ab12848d6 -> 7d0b1c927 [SPARK-21228][SQL][BRANCH-2.2] InSet incorrect handling of structs ## What changes were proposed in this pull request? This is backport of https://github.com/apache/spark/pull/18455 When data type is struct,

spark git commit: [SPARK-20609][MLLIB][TEST] manually cleared 'spark.local.dir' before/after a test in ALSCleanerSuite

2017-07-08 Thread srowen
Repository: spark Updated Branches: refs/heads/master 01f183e84 -> 330bf5c99 [SPARK-20609][MLLIB][TEST] manually cleared 'spark.local.dir' before/after a test in ALSCleanerSuite ## What changes were proposed in this pull request? This PR is similar to #17869. Once` 'spark.local.dir'` is

spark git commit: Mesos doc fixes

2017-07-08 Thread srowen
Repository: spark Updated Branches: refs/heads/master f5f02d213 -> 01f183e84 Mesos doc fixes ## What changes were proposed in this pull request? Some link fixes for the documentation [Running Spark on Mesos](https://spark.apache.org/docs/latest/running-on-mesos.html): * Updated Link to

spark git commit: [SPARK-20456][DOCS] Add examples for functions collection for pyspark

2017-07-08 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master a7b46c627 -> f5f02d213 [SPARK-20456][DOCS] Add examples for functions collection for pyspark ## What changes were proposed in this pull request? This adds documentation to many functions in pyspark.sql.functions.py: `upper`, `lower`,

spark git commit: [SPARK-20307][SPARKR] SparkR: pass on setHandleInvalid to spark.mllib functions that use StringIndexer

2017-07-08 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master d0bfc6733 -> a7b46c627 [SPARK-20307][SPARKR] SparkR: pass on setHandleInvalid to spark.mllib functions that use StringIndexer ## What changes were proposed in this pull request? For randomForest classifier, if test data contains unseen

spark git commit: [SPARK-21069][SS][DOCS] Add rate source to programming guide.

2017-07-08 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master 9760c15ac -> d0bfc6733 [SPARK-21069][SS][DOCS] Add rate source to programming guide. ## What changes were proposed in this pull request? SPARK-20979 added a new structured streaming source: Rate source. This patch adds the corresponding

spark git commit: [SPARK-21069][SS][DOCS] Add rate source to programming guide.

2017-07-08 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-2.2 576fd4c3a -> ab12848d6 [SPARK-21069][SS][DOCS] Add rate source to programming guide. ## What changes were proposed in this pull request? SPARK-20979 added a new structured streaming source: Rate source. This patch adds the

spark git commit: [SPARK-20379][CORE] Allow SSL config to reference env variables.

2017-07-08 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 7896e7b99 -> 9760c15ac [SPARK-20379][CORE] Allow SSL config to reference env variables. This change exposes the internal code path in SparkConf that allows configs to be read with variable substitution applied, and uses that new method in

spark git commit: [SPARK-21281][SQL] Use string types by default if array and map have no argument

2017-07-08 Thread lixiao
Repository: spark Updated Branches: refs/heads/master e1a172c20 -> 7896e7b99 [SPARK-21281][SQL] Use string types by default if array and map have no argument ## What changes were proposed in this pull request? This pr modified code to use string types by default if `array` and `map` in