spark git commit: [SPARK-18339][SPARK-18513][SQL] Don't push down current_timestamp for filters in StructuredStreaming and persist batch and watermark timestamps to offset log.

2016-11-28 Thread tdas
Repository: spark Updated Branches: refs/heads/branch-2.1 27a1a5c99 -> ea6957da2 [SPARK-18339][SPARK-18513][SQL] Don't push down current_timestamp for filters in StructuredStreaming and persist batch and watermark timestamps to offset log. ## What changes were proposed in this pull request?

spark git commit: [SPARK-18339][SPARK-18513][SQL] Don't push down current_timestamp for filters in StructuredStreaming and persist batch and watermark timestamps to offset log.

2016-11-28 Thread tdas
Repository: spark Updated Branches: refs/heads/master e2318ede0 -> 3c0beea47 [SPARK-18339][SPARK-18513][SQL] Don't push down current_timestamp for filters in StructuredStreaming and persist batch and watermark timestamps to offset log. ## What changes were proposed in this pull request? For

spark git commit: [SPARK-18544][SQL] Append with df.saveAsTable writes data to wrong location

2016-11-28 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.1 1759cf69a -> 27a1a5c99 [SPARK-18544][SQL] Append with df.saveAsTable writes data to wrong location ## What changes were proposed in this pull request? We failed to properly propagate table metadata for existing tables for the

spark git commit: [SPARK-18544][SQL] Append with df.saveAsTable writes data to wrong location

2016-11-28 Thread rxin
Repository: spark Updated Branches: refs/heads/master d449988b8 -> e2318ede0 [SPARK-18544][SQL] Append with df.saveAsTable writes data to wrong location ## What changes were proposed in this pull request? We failed to properly propagate table metadata for existing tables for the saveAsTable

spark git commit: [SPARK-18058][SQL][TRIVIAL] Use dataType.sameResult(...) instead equality on asNullable datatypes

2016-11-28 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.1 c4cbdc864 -> 1759cf69a [SPARK-18058][SQL][TRIVIAL] Use dataType.sameResult(...) instead equality on asNullable datatypes ## What changes were proposed in this pull request? This is absolutely minor. PR

spark git commit: [SPARK-18058][SQL][TRIVIAL] Use dataType.sameResult(...) instead equality on asNullable datatypes

2016-11-28 Thread rxin
Repository: spark Updated Branches: refs/heads/master 8b325b17e -> d449988b8 [SPARK-18058][SQL][TRIVIAL] Use dataType.sameResult(...) instead equality on asNullable datatypes ## What changes were proposed in this pull request? This is absolutely minor. PR

spark git commit: [SPARK-18547][CORE] Propagate I/O encryption key when executors register.

2016-11-28 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-2.1 45e2b3c0e -> c4cbdc864 [SPARK-18547][CORE] Propagate I/O encryption key when executors register. This change modifies the method used to propagate encryption keys used during shuffle. Instead of relying on YARN's UserGroupInformation

spark git commit: [SPARK-18547][CORE] Propagate I/O encryption key when executors register.

2016-11-28 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master 1633ff3b6 -> 8b325b17e [SPARK-18547][CORE] Propagate I/O encryption key when executors register. This change modifies the method used to propagate encryption keys used during shuffle. Instead of relying on YARN's UserGroupInformation

spark git commit: [SPARK-18588][SS][KAFKA] Ignore the flaky kafka test

2016-11-28 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-2.1 a0c1c699e -> 45e2b3c0e [SPARK-18588][SS][KAFKA] Ignore the flaky kafka test ## What changes were proposed in this pull request? Ignore the flaky test to unblock other PRs while I'm debugging it. ## How was this patch tested? Jenkins

spark git commit: [SPARK-18588][SS][KAFKA] Ignore the flaky kafka test

2016-11-28 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master e64a2047e -> 1633ff3b6 [SPARK-18588][SS][KAFKA] Ignore the flaky kafka test ## What changes were proposed in this pull request? Ignore the flaky test to unblock other PRs while I'm debugging it. ## How was this patch tested? Jenkins

spark git commit: [SPARK-18523][PYSPARK] Make SparkContext.stop more reliable

2016-11-28 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.1 cdf315ba1 -> c46928ff9 [SPARK-18523][PYSPARK] Make SparkContext.stop more reliable ## What changes were proposed in this pull request? This PR fixes SparkContext broken state in which it may fall if spark driver get crashed or killed

spark git commit: [SPARK-18403][SQL] Fix unsafe data false sharing issue in ObjectHashAggregateExec

2016-11-28 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 05f7c6ffa -> 2e809903d [SPARK-18403][SQL] Fix unsafe data false sharing issue in ObjectHashAggregateExec ## What changes were proposed in this pull request? This PR fixes a random OOM issue occurred while running

spark git commit: [SPARK-18408][ML] API Improvements for LSH

2016-11-28 Thread jkbradley
Repository: spark Updated Branches: refs/heads/branch-2.1 75d73d13e -> cdf315ba1 [SPARK-18408][ML] API Improvements for LSH ## What changes were proposed in this pull request? (1) Change output schema to `Array of Vector` instead of `Vectors` (2) Use `numHashTables` as the dimension of Array

spark git commit: [SPARK-18408][ML] API Improvements for LSH

2016-11-28 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 8b1609beb -> 05f7c6ffa [SPARK-18408][ML] API Improvements for LSH ## What changes were proposed in this pull request? (1) Change output schema to `Array of Vector` instead of `Vectors` (2) Use `numHashTables` as the dimension of Array (3)

spark git commit: [SPARK-18553][CORE][BRANCH-2.0] Fix leak of TaskSetManager following executor loss

2016-11-28 Thread joshrosen
Repository: spark Updated Branches: refs/heads/branch-2.0 f158045fd -> 9ff03fa23 [SPARK-18553][CORE][BRANCH-2.0] Fix leak of TaskSetManager following executor loss ## What changes were proposed in this pull request? This patch fixes a critical resource leak in the TaskScheduler which could

spark git commit: [SPARK-18117][CORE] Add test for TaskSetBlacklist

2016-11-28 Thread irashid
Repository: spark Updated Branches: refs/heads/master ad67993b7 -> 8b1609beb [SPARK-18117][CORE] Add test for TaskSetBlacklist ## What changes were proposed in this pull request? This adds tests to verify the interaction between TaskSetBlacklist and TaskSchedulerImpl. TaskSetBlacklist was

[1/2] spark git commit: Preparing Spark release v2.1.0-rc1

2016-11-28 Thread pwendell
Repository: spark Updated Branches: refs/heads/branch-2.1 b386943b2 -> 75d73d13e Preparing Spark release v2.1.0-rc1 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/80aabc0b Tree:

[2/2] spark git commit: Preparing development version 2.1.1-SNAPSHOT

2016-11-28 Thread pwendell
Preparing development version 2.1.1-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/75d73d13 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/75d73d13 Diff:

[spark] Git Push Summary

2016-11-28 Thread pwendell
Repository: spark Updated Tags: refs/tags/v2.1.0-rc1 [created] 80aabc0bd - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-18597][SQL] Do not push-down join conditions to the left side of a Left Anti join [BRANCH-2.0]

2016-11-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.0 759bd4a6a -> f158045fd [SPARK-18597][SQL] Do not push-down join conditions to the left side of a Left Anti join [BRANCH-2.0] ## What changes were proposed in this pull request? We currently push down join conditions of a Left Anti

spark git commit: [SPARK-17680][SQL][TEST] Added test cases for InMemoryRelation

2016-11-28 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-2.1 81e3f9711 -> b386943b2 [SPARK-17680][SQL][TEST] Added test cases for InMemoryRelation ## What changes were proposed in this pull request? This pull request adds test cases for the following cases: - keep all data types with null or

spark git commit: [SPARK-16282][SQL] Implement percentile SQL function.

2016-11-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.1 4d7947856 -> 81e3f9711 [SPARK-16282][SQL] Implement percentile SQL function. ## What changes were proposed in this pull request? Implement percentile SQL function. It computes the exact percentile(s) of expr at pc with range in [0,

spark git commit: [SPARK-17680][SQL][TEST] Added test cases for InMemoryRelation

2016-11-28 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 0f5f52a3d -> ad67993b7 [SPARK-17680][SQL][TEST] Added test cases for InMemoryRelation ## What changes were proposed in this pull request? This pull request adds test cases for the following cases: - keep all data types with null or

spark git commit: [SPARK-16282][SQL] Implement percentile SQL function.

2016-11-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 185642846 -> 0f5f52a3d [SPARK-16282][SQL] Implement percentile SQL function. ## What changes were proposed in this pull request? Implement percentile SQL function. It computes the exact percentile(s) of expr at pc with range in [0, 1].

spark git commit: [SQL][MINOR] DESC should use 'Catalog' as partition provider

2016-11-28 Thread rxin
Repository: spark Updated Branches: refs/heads/master eba727757 -> 185642846 [SQL][MINOR] DESC should use 'Catalog' as partition provider ## What changes were proposed in this pull request? `CatalogTable` has a parameter named `tracksPartitionsInCatalog`, and in `CatalogTable.toString` we

spark git commit: [SPARK-18602] Set the version of org.codehaus.janino:commons-compiler to 3.0.0 to match the version of org.codehaus.janino:janino

2016-11-28 Thread yhuai
Repository: spark Updated Branches: refs/heads/branch-2.1 32b259fae -> 34ad4d520 [SPARK-18602] Set the version of org.codehaus.janino:commons-compiler to 3.0.0 to match the version of org.codehaus.janino:janino ## What changes were proposed in this pull request? org.codehaus.janino:janino

spark git commit: [SPARK-18602] Set the version of org.codehaus.janino:commons-compiler to 3.0.0 to match the version of org.codehaus.janino:janino

2016-11-28 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 237c3b964 -> eba727757 [SPARK-18602] Set the version of org.codehaus.janino:commons-compiler to 3.0.0 to match the version of org.codehaus.janino:janino ## What changes were proposed in this pull request? org.codehaus.janino:janino

spark git commit: [SPARK-18535][UI][YARN] Redact sensitive information from Spark logs and UI

2016-11-28 Thread vanzin
Repository: spark Updated Branches: refs/heads/master d31ff9b7c -> 237c3b964 [SPARK-18535][UI][YARN] Redact sensitive information from Spark logs and UI ## What changes were proposed in this pull request? This patch adds a new property called `spark.secret.redactionPattern` that allows users

spark git commit: [SPARK-17732][SQL] Revert ALTER TABLE DROP PARTITION should support comparators

2016-11-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 38e29824d -> d31ff9b7c [SPARK-17732][SQL] Revert ALTER TABLE DROP PARTITION should support comparators ## What changes were proposed in this pull request? https://github.com/apache/spark/pull/15704 will fail if we use int literal in

spark git commit: [SPARK-18597][SQL] Do not push-down join conditions to the right side of a LEFT ANTI join

2016-11-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 9f273c517 -> 38e29824d [SPARK-18597][SQL] Do not push-down join conditions to the right side of a LEFT ANTI join ## What changes were proposed in this pull request? We currently push down join conditions of a Left Anti join to both sides

spark git commit: [SPARK-18597][SQL] Do not push-down join conditions to the right side of a LEFT ANTI join

2016-11-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.1 a9d4febe9 -> 32b259fae [SPARK-18597][SQL] Do not push-down join conditions to the right side of a LEFT ANTI join ## What changes were proposed in this pull request? We currently push down join conditions of a Left Anti join to both

spark git commit: [SPARK-17783][SQL] Hide Credentials in CREATE and DESC FORMATTED/EXTENDED a PERSISTENT/TEMP Table for JDBC

2016-11-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.1 e449f7546 -> a9d4febe9 [SPARK-17783][SQL] Hide Credentials in CREATE and DESC FORMATTED/EXTENDED a PERSISTENT/TEMP Table for JDBC ### What changes were proposed in this pull request? We should never expose the Credentials in the

spark git commit: [SPARK-17783][SQL] Hide Credentials in CREATE and DESC FORMATTED/EXTENDED a PERSISTENT/TEMP Table for JDBC

2016-11-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 70dfdcbbf -> 9f273c517 [SPARK-17783][SQL] Hide Credentials in CREATE and DESC FORMATTED/EXTENDED a PERSISTENT/TEMP Table for JDBC ### What changes were proposed in this pull request? We should never expose the Credentials in the EXPLAIN

spark git commit: [SPARK-18118][SQL] fix a compilation error due to nested JavaBeans

2016-11-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.0 9070bd31c -> 759bd4a6a [SPARK-18118][SQL] fix a compilation error due to nested JavaBeans Remove this reference. (cherry picked from commit 70dfdcbbf11c9c3174abc111afa2250236e31af2) Signed-off-by: Herman van Hovell

spark git commit: [SPARK-18118][SQL] fix a compilation error due to nested JavaBeans

2016-11-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.1 712bd5abc -> e449f7546 [SPARK-18118][SQL] fix a compilation error due to nested JavaBeans Remove this reference. (cherry picked from commit 70dfdcbbf11c9c3174abc111afa2250236e31af2) Signed-off-by: Herman van Hovell

spark git commit: [SPARK-18118][SQL] fix a compilation error due to nested JavaBeans\nRemove this reference.

2016-11-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master f075cd9cb -> 70dfdcbbf [SPARK-18118][SQL] fix a compilation error due to nested JavaBeans\nRemove this reference. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-18118][SQL] fix a compilation error due to nested JavaBeans

2016-11-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.1 d6e027e61 -> 712bd5abc [SPARK-18118][SQL] fix a compilation error due to nested JavaBeans ## What changes were proposed in this pull request? This PR avoids a compilation error due to more than 64KB Java byte code size. This error

spark git commit: [SPARK-18118][SQL] fix a compilation error due to nested JavaBeans

2016-11-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.0 e67ce4837 -> 9070bd31c [SPARK-18118][SQL] fix a compilation error due to nested JavaBeans ## What changes were proposed in this pull request? This PR avoids a compilation error due to more than 64KB Java byte code size. This error

spark git commit: [SPARK-18118][SQL] fix a compilation error due to nested JavaBeans

2016-11-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 454b80499 -> f075cd9cb [SPARK-18118][SQL] fix a compilation error due to nested JavaBeans ## What changes were proposed in this pull request? This PR avoids a compilation error due to more than 64KB Java byte code size. This error occur

spark git commit: [SPARK-18604][SQL] Make sure CollapseWindow returns the attributes in the same order.

2016-11-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.1 886f880df -> d6e027e61 [SPARK-18604][SQL] Make sure CollapseWindow returns the attributes in the same order. ## What changes were proposed in this pull request? The `CollapseWindow` optimizer rule changes the order of output

spark git commit: [SPARK-18604][SQL] Make sure CollapseWindow returns the attributes in the same order.

2016-11-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 87141622e -> 454b80499 [SPARK-18604][SQL] Make sure CollapseWindow returns the attributes in the same order. ## What changes were proposed in this pull request? The `CollapseWindow` optimizer rule changes the order of output attributes.