spark git commit: [SPARK-21912][SQL] ORC/Parquet table should not create invalid column names

2017-09-06 Thread lixiao
Repository: spark Updated Branches: refs/heads/master ce7293c15 -> eea2b877c [SPARK-21912][SQL] ORC/Parquet table should not create invalid column names ## What changes were proposed in this pull request? Currently, users meet job abortions while creating or altering ORC/Parquet tables with

spark git commit: [SPARK-21835][SQL][FOLLOW-UP] RewritePredicateSubquery should not produce unresolved query plans

2017-09-06 Thread lixiao
Repository: spark Updated Branches: refs/heads/master aad212547 -> ce7293c15 [SPARK-21835][SQL][FOLLOW-UP] RewritePredicateSubquery should not produce unresolved query plans ## What changes were proposed in this pull request? This is a follow-up of #19050 to deal with `ExistenceJoin` case.

spark git commit: Fixed pandoc dependency issue in python/setup.py

2017-09-06 Thread gurwls223
Repository: spark Updated Branches: refs/heads/branch-2.2 342cc2a4c -> 49968de52 Fixed pandoc dependency issue in python/setup.py ## Problem Description When pyspark is listed as a dependency of another package, installing the other package will cause an install failure in pyspark. When the

spark git commit: Fixed pandoc dependency issue in python/setup.py

2017-09-06 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master fa0092bdd -> aad212547 Fixed pandoc dependency issue in python/setup.py ## Problem Description When pyspark is listed as a dependency of another package, installing the other package will cause an install failure in pyspark. When the

spark git commit: [SPARK-21901][SS] Define toString for StateOperatorProgress

2017-09-06 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-2.2 9afab9a52 -> 342cc2a4c [SPARK-21901][SS] Define toString for StateOperatorProgress ## What changes were proposed in this pull request? Just `StateOperatorProgress.toString` + few formatting fixes ## How was this patch tested? Local

spark git commit: [SPARK-21901][SS] Define toString for StateOperatorProgress

2017-09-06 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master acdf45fb5 -> fa0092bdd [SPARK-21901][SS] Define toString for StateOperatorProgress ## What changes were proposed in this pull request? Just `StateOperatorProgress.toString` + few formatting fixes ## How was this patch tested? Local

spark git commit: [SPARK-21765] Check that optimization doesn't affect isStreaming bit.

2017-09-06 Thread tdas
Repository: spark Updated Branches: refs/heads/master 36b48ee6e -> acdf45fb5 [SPARK-21765] Check that optimization doesn't affect isStreaming bit. ## What changes were proposed in this pull request? Add an assert in logical plan optimization that the isStreaming bit stays the same, and fix

spark git commit: [SPARK-21801][SPARKR][TEST] set random seed for predictable test

2017-09-06 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master f2e22aebf -> 36b48ee6e [SPARK-21801][SPARKR][TEST] set random seed for predictable test ## What changes were proposed in this pull request? set.seed() before running tests ## How was this patch tested? jenkins, appveyor Author: Felix

spark git commit: [SPARK-21835][SQL] RewritePredicateSubquery should not produce unresolved query plans

2017-09-06 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 64936c14a -> f2e22aebf [SPARK-21835][SQL] RewritePredicateSubquery should not produce unresolved query plans ## What changes were proposed in this pull request? Correlated predicate subqueries are rewritten into `Join` by the rule

spark git commit: [SPARK-21903][BUILD][FOLLOWUP] Upgrade scalastyle-maven-plugin and scalastyle as well in POM and SparkBuild.scala

2017-09-06 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 16c4c03c7 -> 64936c14a [SPARK-21903][BUILD][FOLLOWUP] Upgrade scalastyle-maven-plugin and scalastyle as well in POM and SparkBuild.scala ## What changes were proposed in this pull request? This PR proposes to match scalastyle version in

spark git commit: [SPARK-19357][ML] Adding parallel model evaluation in ML tuning

2017-09-06 Thread mlnick
Repository: spark Updated Branches: refs/heads/master 4ee7dfe41 -> 16c4c03c7 [SPARK-19357][ML] Adding parallel model evaluation in ML tuning ## What changes were proposed in this pull request? Modified `CrossValidator` and `TrainValidationSplit` to be able to evaluate models in parallel for

spark git commit: [SPARK-21924][DOCS] Update structured streaming programming guide doc

2017-09-06 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 7da8fbf08 -> 9afab9a52 [SPARK-21924][DOCS] Update structured streaming programming guide doc ## What changes were proposed in this pull request? Update the line "For example, the data (12:09, cat) is out of order and late, and it

spark git commit: [SPARK-21924][DOCS] Update structured streaming programming guide doc

2017-09-06 Thread srowen
Repository: spark Updated Branches: refs/heads/master 445f1790a -> 4ee7dfe41 [SPARK-21924][DOCS] Update structured streaming programming guide doc ## What changes were proposed in this pull request? Update the line "For example, the data (12:09, cat) is out of order and late, and it falls