spark git commit: [SPARK-21789][PYTHON] Remove obsolete codes for parsing abstract schema strings

2017-08-31 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 5cd8ea99f -> 648a8626b [SPARK-21789][PYTHON] Remove obsolete codes for parsing abstract schema strings ## What changes were proposed in this pull request? This PR proposes to remove private functions that look not used in the main codes,

spark git commit: [SPARK-21779][PYTHON] Simpler DataFrame.sample API in Python

2017-08-31 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master f5e10a34e -> 5cd8ea99f [SPARK-21779][PYTHON] Simpler DataFrame.sample API in Python ## What changes were proposed in this pull request? This PR make `DataFrame.sample(...)` can omit `withReplacement` defaulting `False`, consistently with

spark git commit: [SPARK-21862][ML] Add overflow check in PCA

2017-08-31 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 96028e36b -> f5e10a34e [SPARK-21862][ML] Add overflow check in PCA ## What changes were proposed in this pull request? add overflow check in PCA, otherwise it is possible to throw `NegativeArraySizeException` when `k` and `numFeatures`

spark git commit: [SPARK-17139][ML][FOLLOW-UP] Add convenient method `asBinary` for casting to BinaryLogisticRegressionSummary

2017-08-31 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master cba69aeb4 -> 96028e36b [SPARK-17139][ML][FOLLOW-UP] Add convenient method `asBinary` for casting to BinaryLogisticRegressionSummary ## What changes were proposed in this pull request? add an "asBinary" method to LogisticRegressionSummary

spark git commit: [SPARK-21110][SQL] Structs, arrays, and other orderable datatypes should be usable in inequalities

2017-08-31 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 7ce110828 -> cba69aeb4 [SPARK-21110][SQL] Structs, arrays, and other orderable datatypes should be usable in inequalities ## What changes were proposed in this pull request? Allows `BinaryComparison` operators to work on any data type

spark git commit: [SPARK-17107][SQL][FOLLOW-UP] Remove redundant pushdown rule for Union

2017-08-31 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 501370d9d -> 7ce110828 [SPARK-17107][SQL][FOLLOW-UP] Remove redundant pushdown rule for Union ## What changes were proposed in this pull request? Also remove useless function `partitionByDeterministic` after the changes of

spark git commit: [SPARK-21583][HOTFIX] Removed intercept in test causing failures

2017-08-31 Thread lixiao
Repository: spark Updated Branches: refs/heads/master fc45c2c88 -> 501370d9d [SPARK-21583][HOTFIX] Removed intercept in test causing failures Removing a check in the ColumnarBatchSuite that depended on a Java assertion. This assertion is being compiled out in the Maven builds causing the

spark git commit: [SPARK-20812][MESOS] Add secrets support to the dispatcher

2017-08-31 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 9696580c3 -> fc45c2c88 [SPARK-20812][MESOS] Add secrets support to the dispatcher Mesos has secrets primitives for environment and file-based secrets, this PR adds that functionality to the Spark dispatcher and the appropriate

spark git commit: [SPARK-21886][SQL] Use SparkSession.internalCreateDataFrame to create…

2017-08-31 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 19b0240d4 -> 9696580c3 [SPARK-21886][SQL] Use SparkSession.internalCreateDataFrame to create… … Dataset with LogicalRDD logical operator ## What changes were proposed in this pull request? Reusing

spark git commit: [SPARK-21878][SQL][TEST] Create SQLMetricsTestUtils

2017-08-31 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 964b507c7 -> 19b0240d4 [SPARK-21878][SQL][TEST] Create SQLMetricsTestUtils ## What changes were proposed in this pull request? Creates `SQLMetricsTestUtils` for the utility functions of both Hive-specific and the other SQLMetrics test