spark git commit: [SPARK-20040][ML][PYTHON] pyspark wrapper for ChiSquareTest

2017-03-28 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 7d432af8f -> a5c87707e [SPARK-20040][ML][PYTHON] pyspark wrapper for ChiSquareTest ## What changes were proposed in this pull request? A pyspark wrapper for spark.ml.stat.ChiSquareTest ## How was this patch tested? unit tests doctests

spark git commit: [SPARK-20043][ML] DecisionTreeModel: ImpurityCalculator builder fails for uppercase impurity type Gini

2017-03-28 Thread jkbradley
Repository: spark Updated Branches: refs/heads/branch-2.1 4964dbedb -> 30954806f [SPARK-20043][ML] DecisionTreeModel: ImpurityCalculator builder fails for uppercase impurity type Gini Fix bug: DecisionTreeModel can't recongnize Impurity "Gini" when loading TODO: + [x] add unit test + [x]

spark git commit: [SPARK-20043][ML] DecisionTreeModel: ImpurityCalculator builder fails for uppercase impurity type Gini

2017-03-28 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 92e385e0b -> 7d432af8f [SPARK-20043][ML] DecisionTreeModel: ImpurityCalculator builder fails for uppercase impurity type Gini Fix bug: DecisionTreeModel can't recongnize Impurity "Gini" when loading TODO: + [x] add unit test + [x] fix

[1/2] spark git commit: Preparing Spark release v2.1.1-rc2

2017-03-28 Thread pwendell
Repository: spark Updated Branches: refs/heads/branch-2.1 e669dd7ea -> 4964dbedb Preparing Spark release v2.1.1-rc2 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/02b165dc Tree:

[2/2] spark git commit: Preparing development version 2.1.2-SNAPSHOT

2017-03-28 Thread pwendell
Preparing development version 2.1.2-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4964dbed Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/4964dbed Diff:

[spark] Git Push Summary

2017-03-28 Thread pwendell
Repository: spark Updated Tags: refs/tags/v2.1.1-rc2 [created] 02b165dcc - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-14536][SQL][BACKPORT-2.1] fix to handle null value in array type column for postgres.

2017-03-28 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.1 fd2e40614 -> e669dd7ea [SPARK-14536][SQL][BACKPORT-2.1] fix to handle null value in array type column for postgres. ## What changes were proposed in this pull request? JDBC read is failing with NPE due to missing null value check for

spark git commit: [SPARK-20125][SQL] Dataset of type option of map does not work

2017-03-28 Thread lian
Repository: spark Updated Branches: refs/heads/branch-2.1 4bcb7d676 -> fd2e40614 [SPARK-20125][SQL] Dataset of type option of map does not work When we build the deserializer expression for map type, we will use `StaticInvoke` to call `ArrayBasedMapData.toScalaMap`, and declare the return

spark git commit: [SPARK-19868] conflict TasksetManager lead to spark stopped

2017-03-28 Thread kayousterhout
Repository: spark Updated Branches: refs/heads/master d4fac410e -> 92e385e0b [SPARK-19868] conflict TasksetManager lead to spark stopped ## What changes were proposed in this pull request? We must set the taskset to zombie before the DAGScheduler handles the taskEnded event. It's possible

spark git commit: [SPARK-20125][SQL] Dataset of type option of map does not work

2017-03-28 Thread lian
Repository: spark Updated Branches: refs/heads/master 17eddb35a -> d4fac410e [SPARK-20125][SQL] Dataset of type option of map does not work ## What changes were proposed in this pull request? When we build the deserializer expression for map type, we will use `StaticInvoke` to call

spark git commit: [SPARK-19995][YARN] Register tokens to current UGI to avoid re-issuing of tokens in yarn client mode

2017-03-28 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.1 4056191d3 -> 4bcb7d676 [SPARK-19995][YARN] Register tokens to current UGI to avoid re-issuing of tokens in yarn client mode ## What changes were proposed in this pull request? In the current Spark on YARN code, we will obtain tokens

spark git commit: [SPARK-19995][YARN] Register tokens to current UGI to avoid re-issuing of tokens in yarn client mode

2017-03-28 Thread vanzin
Repository: spark Updated Branches: refs/heads/master f82461fc1 -> 17eddb35a [SPARK-19995][YARN] Register tokens to current UGI to avoid re-issuing of tokens in yarn client mode ## What changes were proposed in this pull request? In the current Spark on YARN code, we will obtain tokens from

spark git commit: [SPARK-20126][SQL] Remove HiveSessionState

2017-03-28 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 4fcc214d9 -> f82461fc1 [SPARK-20126][SQL] Remove HiveSessionState ## What changes were proposed in this pull request? Commit https://github.com/apache/spark/commit/ea361165e1ddce4d8aa0242ae3e878d7b39f1de2 moved most of the logic from the

spark git commit: [SPARK-20124][SQL] Join reorder should keep the same order of final project attributes

2017-03-28 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 91559d277 -> 4fcc214d9 [SPARK-20124][SQL] Join reorder should keep the same order of final project attributes ## What changes were proposed in this pull request? Join reorder algorithm should keep exactly the same order of output

spark git commit: [SPARK-20094][SQL] Preventing push down of IN subquery to Join operator

2017-03-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master a9abff281 -> 91559d277 [SPARK-20094][SQL] Preventing push down of IN subquery to Join operator ## What changes were proposed in this pull request? TPCDS q45 fails becuase: `ReorderJoin` collects all predicates and try to put them into

spark git commit: [SPARK-20119][TEST-MAVEN] Fix the test case fail in DataSourceScanExecRedactionSuite

2017-03-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 6c70a38c2 -> a9abff281 [SPARK-20119][TEST-MAVEN] Fix the test case fail in DataSourceScanExecRedactionSuite ### What changes were proposed in this pull request? Changed the pattern to match the first n characters in the location field so