spark git commit: [SPARK-13652][CORE] Copy ByteBuffer in sendRpcSync as it will be recycled

2016-03-03 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-1.6 b3a512965 -> 51c676e46 [SPARK-13652][CORE] Copy ByteBuffer in sendRpcSync as it will be recycled ## What changes were proposed in this pull request? `sendRpcSync` should copy the response content because the underlying buffer will be

spark git commit: [SPARK-13652][CORE] Copy ByteBuffer in sendRpcSync as it will be recycled

2016-03-03 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master f6ac7c30d -> 465c665db [SPARK-13652][CORE] Copy ByteBuffer in sendRpcSync as it will be recycled ## What changes were proposed in this pull request? `sendRpcSync` should copy the response content because the underlying buffer will be recy

spark git commit: [SPARK-12941][SQL][MASTER] Spark-SQL JDBC Oracle dialect fails to map string datatypes to Oracle VARCHAR datatype mapping

2016-03-03 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 15d57f9c2 -> f6ac7c30d [SPARK-12941][SQL][MASTER] Spark-SQL JDBC Oracle dialect fails to map string datatypes to Oracle VARCHAR datatype mapping ## What changes were proposed in this pull request? A test suite added for the bug fix -SPARK

spark git commit: [SPARK-13647] [SQL] also check if numeric value is within allowed range in _verify_type

2016-03-03 Thread davies
Repository: spark Updated Branches: refs/heads/master d062587dd -> 15d57f9c2 [SPARK-13647] [SQL] also check if numeric value is within allowed range in _verify_type ## What changes were proposed in this pull request? This PR makes the `_verify_type` in `types.py` more strict, also check if

spark git commit: [SPARK-13601] [TESTS] use 1 partition in tests to avoid race conditions

2016-03-03 Thread davies
Repository: spark Updated Branches: refs/heads/branch-1.6 fa86dc47a -> b3a512965 [SPARK-13601] [TESTS] use 1 partition in tests to avoid race conditions Fix race conditions when cleanup files. Existing tests. Author: Davies Liu Closes #11507 from davies/flaky. (cherry picked from commit d

spark git commit: [SPARK-13601] [TESTS] use 1 partition in tests to avoid race conditions

2016-03-03 Thread davies
Repository: spark Updated Branches: refs/heads/master b373a8886 -> d062587dd [SPARK-13601] [TESTS] use 1 partition in tests to avoid race conditions ## What changes were proposed in this pull request? Fix race conditions when cleanup files. ## How was this patch tested? Existing tests. Aut

spark git commit: [SPARK-13415][SQL] Visualize subquery in SQL web UI

2016-03-03 Thread yhuai
Repository: spark Updated Branches: refs/heads/master ad0de99f3 -> b373a8886 [SPARK-13415][SQL] Visualize subquery in SQL web UI ## What changes were proposed in this pull request? This PR support visualization for subquery in SQL web UI, also improve the explain of subquery, especially when

spark git commit: [SPARK-13584][SQL][TESTS] Make ContinuousQueryManagerSuite not output logs to the console

2016-03-03 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master 3edcc4022 -> ad0de99f3 [SPARK-13584][SQL][TESTS] Make ContinuousQueryManagerSuite not output logs to the console ## What changes were proposed in this pull request? Make ContinuousQueryManagerSuite not output logs to the console. The logs

spark git commit: [SPARK-13632][SQL] Move commands.scala to command package

2016-03-03 Thread rxin
Repository: spark Updated Branches: refs/heads/master 941b270b7 -> 3edcc4022 [SPARK-13632][SQL] Move commands.scala to command package ## What changes were proposed in this pull request? This patch simply moves things to a new package in an effort to reduce the size of the diff in #11048. Cu

spark git commit: [MINOR] Fix typos in comments and testcase name of code

2016-03-03 Thread srowen
Repository: spark Updated Branches: refs/heads/master 52035d103 -> 941b270b7 [MINOR] Fix typos in comments and testcase name of code ## What changes were proposed in this pull request? This PR fixes typos in comments and testcase name of code. ## How was this patch tested? manual. Author:

spark git commit: [SPARK-13423][HOTFIX] Static analysis fixes for 2.x / fixed for Scala 2.10, again

2016-03-03 Thread srowen
Repository: spark Updated Branches: refs/heads/master ce58e99aa -> 52035d103 [SPARK-13423][HOTFIX] Static analysis fixes for 2.x / fixed for Scala 2.10, again ## What changes were proposed in this pull request? Fixes (another) compile problem due to inadvertent use of Option.contains, only

spark git commit: [MINOR][ML][DOC] Remove duplicated periods at the end of some sharedParam

2016-03-03 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master cf95d728c -> ce58e99aa [MINOR][ML][DOC] Remove duplicated periods at the end of some sharedParam ## What changes were proposed in this pull request? Remove duplicated periods at the end of some sharedParams in ScalaDoc, such as [here](http

spark git commit: [SPARK-13543][SQL] Support for specifying compression codec for Parquet/ORC via option()

2016-03-03 Thread rxin
Repository: spark Updated Branches: refs/heads/master 511d4929c -> cf95d728c [SPARK-13543][SQL] Support for specifying compression codec for Parquet/ORC via option() ## What changes were proposed in this pull request? This PR adds the support to specify compression codecs for both ORC and Pa

spark git commit: [SPARK-13601] call failure callbacks before writer.close()

2016-03-03 Thread davies
Repository: spark Updated Branches: refs/heads/branch-1.6 1ce2c1235 -> fa86dc47a [SPARK-13601] call failure callbacks before writer.close() In order to tell OutputStream that the task has failed or not, we should call the failure callbacks BEFORE calling writer.close(). Added new unit tests.

spark git commit: [SPARK-12877][ML] Add train-validation-split to pyspark

2016-03-03 Thread meng
Repository: spark Updated Branches: refs/heads/master 9a48c656e -> 511d4929c [SPARK-12877][ML] Add train-validation-split to pyspark ## What changes were proposed in this pull request? The changes proposed were to add train-validation-split to pyspark.ml.tuning. ## How was the this patch test

spark git commit: [SPARK-13599][BUILD] remove transitive groovy dependencies from Hive

2016-03-03 Thread rxin
Repository: spark Updated Branches: refs/heads/master 70f6f9649 -> 9a48c656e [SPARK-13599][BUILD] remove transitive groovy dependencies from Hive ## What changes were proposed in this pull request? Modifies the dependency declarations of the all the hive artifacts, to explicitly exclude the

spark git commit: [SPARK-13013][DOCS] Replace example code in mllib-clustering.md using include_example

2016-03-03 Thread meng
Repository: spark Updated Branches: refs/heads/master 645c3a85e -> 70f6f9649 [SPARK-13013][DOCS] Replace example code in mllib-clustering.md using include_example Replace example code in mllib-clustering.md using include_example https://issues.apache.org/jira/browse/SPARK-13013 The example c

spark git commit: [SPARK-13465] Add a task failure listener to TaskContext

2016-03-03 Thread davies
Repository: spark Updated Branches: refs/heads/branch-1.6 fedb81360 -> 1ce2c1235 [SPARK-13465] Add a task failure listener to TaskContext ## What changes were proposed in this pull request? TaskContext supports task completion callback, which gets called regardless of task failures. However,

spark git commit: [SPARK-13423][HOTFIX] Static analysis fixes for 2.x / fixed for Scala 2.10

2016-03-03 Thread srowen
Repository: spark Updated Branches: refs/heads/master b5f02d674 -> 645c3a85e [SPARK-13423][HOTFIX] Static analysis fixes for 2.x / fixed for Scala 2.10 ## What changes were proposed in this pull request? Fixes compile problem due to inadvertent use of `Option.contains`, only in Scala 2.11. T

[2/3] spark git commit: [SPARK-13583][CORE][STREAMING] Remove unused imports and add checkstyle rule

2016-03-03 Thread srowen
http://git-wip-us.apache.org/repos/asf/spark/blob/b5f02d67/examples/src/main/scala/org/apache/spark/examples/streaming/ActorWordCount.scala -- diff --git a/examples/src/main/scala/org/apache/spark/examples/streaming/ActorWordCount

[1/3] spark git commit: [SPARK-13583][CORE][STREAMING] Remove unused imports and add checkstyle rule

2016-03-03 Thread srowen
Repository: spark Updated Branches: refs/heads/master e97fc7f17 -> b5f02d674 http://git-wip-us.apache.org/repos/asf/spark/blob/b5f02d67/sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala -- diff --git a/sql/core

[3/3] spark git commit: [SPARK-13583][CORE][STREAMING] Remove unused imports and add checkstyle rule

2016-03-03 Thread srowen
[SPARK-13583][CORE][STREAMING] Remove unused imports and add checkstyle rule ## What changes were proposed in this pull request? After SPARK-6990, `dev/lint-java` keeps Java code healthy and helps PR review by saving much time. This issue aims remove unused imports from Java/Scala code and add

[3/3] spark git commit: [SPARK-13423][WIP][CORE][SQL][STREAMING] Static analysis fixes for 2.x

2016-03-03 Thread srowen
[SPARK-13423][WIP][CORE][SQL][STREAMING] Static analysis fixes for 2.x ## What changes were proposed in this pull request? Make some cross-cutting code improvements according to static analysis. These are individually up for discussion since they exist in separate commits that can be reverted.

[1/3] spark git commit: [SPARK-13423][WIP][CORE][SQL][STREAMING] Static analysis fixes for 2.x

2016-03-03 Thread srowen
Repository: spark Updated Branches: refs/heads/master 02b7677e9 -> e97fc7f17 http://git-wip-us.apache.org/repos/asf/spark/blob/e97fc7f1/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala -- di

[2/3] spark git commit: [SPARK-13423][WIP][CORE][SQL][STREAMING] Static analysis fixes for 2.x

2016-03-03 Thread srowen
http://git-wip-us.apache.org/repos/asf/spark/blob/e97fc7f1/examples/src/main/scala/org/apache/spark/examples/streaming/clickstream/PageViewStream.scala -- diff --git a/examples/src/main/scala/org/apache/spark/examples/streaming/cl

spark git commit: [HOT-FIX] Recover some deprecations for 2.10 compatibility.

2016-03-03 Thread srowen
Repository: spark Updated Branches: refs/heads/master 7b25dc7b7 -> 02b7677e9 [HOT-FIX] Recover some deprecations for 2.10 compatibility. ## What changes were proposed in this pull request? #11479 [SPARK-13627] broke 2.10 compatibility: [2.10-Build](https://amplab.cs.berkeley.edu/jenkins/view

spark git commit: [SPARK-13466] [SQL] Remove projects that become redundant after column pruning rule

2016-03-03 Thread davies
Repository: spark Updated Branches: refs/heads/master 1085bd862 -> 7b25dc7b7 [SPARK-13466] [SQL] Remove projects that become redundant after column pruning rule JIRA: https://issues.apache.org/jira/browse/SPARK-13466 ## What changes were proposed in this pull request? With column pruning ru