spark git commit: [SPARK-18001][DOCUMENT] fix broke link to SparkDataFrame

2016-10-18 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 3796a98cf -> cdd2570e6 [SPARK-18001][DOCUMENT] fix broke link to SparkDataFrame ## What changes were proposed in this pull request? In http://spark.apache.org/docs/latest/sql-programming-guide.html, Section "Untyped Dataset

spark git commit: [SPARK-18001][DOCUMENT] fix broke link to SparkDataFrame

2016-10-18 Thread rxin
Repository: spark Updated Branches: refs/heads/master 4329c5cea -> f39852e59 [SPARK-18001][DOCUMENT] fix broke link to SparkDataFrame ## What changes were proposed in this pull request? In http://spark.apache.org/docs/latest/sql-programming-guide.html, Section "Untyped Dataset Operations

spark git commit: [SPARK-17873][SQL] ALTER TABLE RENAME TO should allow users to specify database in destination table name(but have to be same as source table)

2016-10-18 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 2629cd746 -> 4329c5cea [SPARK-17873][SQL] ALTER TABLE RENAME TO should allow users to specify database in destination table name(but have to be same as source table) ## What changes were proposed in this pull request? Unlike Hive, in

spark git commit: [SPARK-17711][TEST-HADOOP2.2] Fix hadoop2.2 compilation error

2016-10-18 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master 5f20ae039 -> 2629cd746 [SPARK-17711][TEST-HADOOP2.2] Fix hadoop2.2 compilation error ## What changes were proposed in this pull request? Fix hadoop2.2 compilation error. ## How was this patch tested? Existing tests. cc tdas zsxwing

spark git commit: [SPARK-17711][TEST-HADOOP2.2] Fix hadoop2.2 compilation error

2016-10-18 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-2.0 99943bf69 -> 3796a98cf [SPARK-17711][TEST-HADOOP2.2] Fix hadoop2.2 compilation error ## What changes were proposed in this pull request? Fix hadoop2.2 compilation error. ## How was this patch tested? Existing tests. cc tdas zsxwing

spark git commit: [SPARK-17980][SQL] Fix refreshByPath for converted Hive tables

2016-10-18 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 941b3f9ac -> 5f20ae039 [SPARK-17980][SQL] Fix refreshByPath for converted Hive tables ## What changes were proposed in this pull request? There was a bug introduced in https://github.com/apache/spark/pull/14690 which broke refreshByPath

spark git commit: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored StreamingQueryListener APIs

2016-10-18 Thread tdas
Repository: spark Updated Branches: refs/heads/master 1e35e9693 -> 941b3f9ac [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored StreamingQueryListener APIs ## What changes were proposed in this pull request? As per rxin request, here are further API changes - Changed

spark git commit: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored StreamingQueryListener APIs for branch-2.0

2016-10-18 Thread tdas
Repository: spark Updated Branches: refs/heads/branch-2.0 f6b87939c -> 99943bf69 [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored StreamingQueryListener APIs for branch-2.0 This is the branch-2.0 PR of #15530 to make the APIs consistent with the master. Since these APIs are experimental

spark git commit: [SPARK-17817] [PYSPARK] [FOLLOWUP] PySpark RDD Repartitioning Results in Highly Skewed Partition Sizes

2016-10-18 Thread davies
Repository: spark Updated Branches: refs/heads/master cd106b050 -> 1e35e9693 [SPARK-17817] [PYSPARK] [FOLLOWUP] PySpark RDD Repartitioning Results in Highly Skewed Partition Sizes ## What changes were proposed in this pull request? This change is a followup for #15389 which calls

spark git commit: [SPARK-17841][STREAMING][KAFKA] drain commitQueue

2016-10-18 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 6ef923137 -> f6b87939c [SPARK-17841][STREAMING][KAFKA] drain commitQueue ## What changes were proposed in this pull request? Actually drain commit queue rather than just iterating it. iterator() on a concurrent linked queue won't

spark git commit: [SPARK-17841][STREAMING][KAFKA] drain commitQueue

2016-10-18 Thread rxin
Repository: spark Updated Branches: refs/heads/master cd662bc7a -> cd106b050 [SPARK-17841][STREAMING][KAFKA] drain commitQueue ## What changes were proposed in this pull request? Actually drain commit queue rather than just iterating it. iterator() on a concurrent linked queue won't remove

spark git commit: Revert "[SPARK-17985][CORE] Bump commons-lang3 version to 3.5."

2016-10-18 Thread rxin
Repository: spark Updated Branches: refs/heads/master b3130c7b6 -> cd662bc7a Revert "[SPARK-17985][CORE] Bump commons-lang3 version to 3.5." This reverts commit bfe7885aee2f406c1bbde08e30809a0b4bb070d2. The commit caused build failures on Hadoop 2.2 profile: ``` [error]

spark git commit: [SPARK-17955][SQL] Make DataFrameReader.jdbc call DataFrameReader.format("jdbc").load

2016-10-18 Thread rxin
Repository: spark Updated Branches: refs/heads/master 4518642ab -> b3130c7b6 [SPARK-17955][SQL] Make DataFrameReader.jdbc call DataFrameReader.format("jdbc").load ## What changes were proposed in this pull request? This PR proposes to make `DataFrameReader.jdbc` call

spark git commit: [SPARK-17930][CORE] The SerializerInstance instance used when deserializing a TaskResult is not reused

2016-10-18 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master 20dd11096 -> 4518642ab [SPARK-17930][CORE] The SerializerInstance instance used when deserializing a TaskResult is not reused ## What changes were proposed in this pull request? The following code is called when the DirectTaskResult

spark git commit: [MINOR][DOC] Add more built-in sources in sql-programming-guide.md

2016-10-18 Thread rxin
Repository: spark Updated Branches: refs/heads/master bfe7885ae -> 20dd11096 [MINOR][DOC] Add more built-in sources in sql-programming-guide.md ## What changes were proposed in this pull request? Add more built-in sources in sql-programming-guide.md. ## How was this patch tested? Manually.

spark git commit: [MINOR][DOC] Add more built-in sources in sql-programming-guide.md

2016-10-18 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 26e978a93 -> 6ef923137 [MINOR][DOC] Add more built-in sources in sql-programming-guide.md ## What changes were proposed in this pull request? Add more built-in sources in sql-programming-guide.md. ## How was this patch tested?

spark git commit: [SPARK-17985][CORE] Bump commons-lang3 version to 3.5.

2016-10-18 Thread rxin
Repository: spark Updated Branches: refs/heads/master 4ef39c2f4 -> bfe7885ae [SPARK-17985][CORE] Bump commons-lang3 version to 3.5. ## What changes were proposed in this pull request? `SerializationUtils.clone()` of commons-lang3 (<3.5) has a bug that breaks thread safety, which gets stack

spark git commit: [SPARK-17711] Compress rolled executor log

2016-10-18 Thread tdas
Repository: spark Updated Branches: refs/heads/branch-2.0 2aa25833c -> 26e978a93 [SPARK-17711] Compress rolled executor log ## What changes were proposed in this pull request? This PR adds support for executor log compression. ## How was this patch tested? Unit tests cc: yhuai tdas mengxr

spark git commit: [SPARK-17711] Compress rolled executor log

2016-10-18 Thread tdas
Repository: spark Updated Branches: refs/heads/master 37686539f -> 231f39e3f [SPARK-17711] Compress rolled executor log ## What changes were proposed in this pull request? This PR adds support for executor log compression. ## How was this patch tested? Unit tests cc: yhuai tdas mengxr

spark git commit: [SPARK-17388] [SQL] Support for inferring type date/timestamp/decimal for partition column

2016-10-18 Thread davies
Repository: spark Updated Branches: refs/heads/master e59df62e6 -> 37686539f [SPARK-17388] [SQL] Support for inferring type date/timestamp/decimal for partition column ## What changes were proposed in this pull request? Currently, Spark only supports to infer `IntegerType`, `LongType`,

spark git commit: [SPARK-17899][SQL][FOLLOW-UP] debug mode should work for corrupted table

2016-10-18 Thread rxin
Repository: spark Updated Branches: refs/heads/master a9e79a41e -> e59df62e6 [SPARK-17899][SQL][FOLLOW-UP] debug mode should work for corrupted table ## What changes were proposed in this pull request? Debug mode should work for corrupted table, so that we can really debug ## How was this

spark git commit: [SQL][STREAMING][TEST] Follow up to remove Option.contains for Scala 2.10 compatibility

2016-10-18 Thread tdas
Repository: spark Updated Branches: refs/heads/branch-2.0 01520de6b -> 9e806f2a4 [SQL][STREAMING][TEST] Follow up to remove Option.contains for Scala 2.10 compatibility ## What changes were proposed in this pull request? Scala 2.10 does not have Option.contains, which broke Scala 2.10

spark git commit: [SQL][STREAMING][TEST] Follow up to remove Option.contains for Scala 2.10 compatibility

2016-10-18 Thread tdas
Repository: spark Updated Branches: refs/heads/master 7d878cf2d -> a9e79a41e [SQL][STREAMING][TEST] Follow up to remove Option.contains for Scala 2.10 compatibility ## What changes were proposed in this pull request? Scala 2.10 does not have Option.contains, which broke Scala 2.10 build.

spark git commit: [SQL][STREAMING][TEST] Fix flaky tests in StreamingQueryListenerSuite

2016-10-18 Thread tdas
Repository: spark Updated Branches: refs/heads/master 1c5a7d7f6 -> 7d878cf2d [SQL][STREAMING][TEST] Fix flaky tests in StreamingQueryListenerSuite This work has largely been done by lw-lin in his PR #15497. This is a slight refactoring of it. ## What changes were proposed in this pull

spark git commit: [SQL][STREAMING][TEST] Fix flaky tests in StreamingQueryListenerSuite

2016-10-18 Thread tdas
Repository: spark Updated Branches: refs/heads/branch-2.0 881e0eb05 -> 01520de6b [SQL][STREAMING][TEST] Fix flaky tests in StreamingQueryListenerSuite This work has largely been done by lw-lin in his PR #15497. This is a slight refactoring of it. ## What changes were proposed in this pull